diff --git a/docs/deploy.md b/docs/deploy.md index 9be372b..d3f4f0e 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -134,8 +134,20 @@ curl -H "x-admin-secret: $ADMIN_REVALIDATE_SECRET" \ ``` It exports every Postgres table to a private blob and sweeps photos that were uploaded but -never attached to anything. It needs `BLOB_READ_WRITE_TOKEN` and refuses without it — the -old `/api/admin/backup` route it replaces dumped Notion and is no longer scheduled. +never attached to anything. It needs a Blob store and refuses without one — the old +`/api/admin/backup` route it replaces dumped Notion and is no longer scheduled. + +**Blob on a laptop.** With no `BLOB_READ_WRITE_TOKEN`, `npm run dev` does not refuse +uploads: every Blob read and write goes to `v5/.blob-data/` (git-ignored), a folder that +behaves like the real store — private and public files, random upload pathnames, copies to +public, list and delete. Photo uploads, chat photos, pending-tool photo promotion, +archived manual PDFs, this backup and the orphan sweep all work. Public files are served +by `GET /api/dev-blob/[...path]` at `AUTH_BASE_URL` (default `http://localhost:3000`); +private ones are never served. Set `BLOB_LOCAL_DISABLE=1` to get the old "uploads are +unavailable" behaviour back. The rule lives in `v5/src/lib/blob-mode.ts`: a token means +Vercel Blob; on Vercel (`VERCEL`) or in a production build without one there is **no** +store and no disk fallback, and `/api/dev-blob/…` answers 404. The Notion import still +insists on a real token, because its rows go to a shared database. Two tables are held out of the file on purpose: `session` and `verification` are sign-in credentials, not records, and the Google tokens on `account` are blanked. A backup is diff --git a/docs/specs/2026-09-14-v5-data-platform-design.md b/docs/specs/2026-09-14-v5-data-platform-design.md index 14f5520..d809759 100644 --- a/docs/specs/2026-09-14-v5-data-platform-design.md +++ b/docs/specs/2026-09-14-v5-data-platform-design.md @@ -1516,7 +1516,6 @@ seed still has no draft or archived tool, so no E2E exercises publish, archive o browser; each is covered against PGlite in `src/app/admin/inventory/actions.test.ts`. **Status.** Accepted. -||||||| 57d7ea3 ### 2026-09-22 — `AUTH_ALLOWED_EMAILS`, so the domain rule cannot become a lock-out @@ -1636,4 +1635,95 @@ that `gallery.spec.ts`'s count would see. reconnecting the mirror. **Status.** Accepted. -||||||| 927bbec + +### 2026-09-23 — Phase 8 built (the Notion mirror), with as-built details + +**What changed.** §3.8 is implemented. `/admin/mirror` belongs to the signed-in admin and has +four parts: `MirrorConnect`, `MirrorMapping`, `MirrorStatus` and `MirrorControls`. **Create +databases** builds all seven databases with fixed schemas; pasted database ids are validated +against those schemas before they are saved. The push lives in `src/lib/mirror/*` and runs as +the `mirrorPush` workflow, in up to six 45 s rounds, so a first sync of the whole inventory makes +progress round by round. `requestMirrorPush()` coalesces bursts of changes and never throws. The +triggers are approval, every tool-editor write, project publish and unpublish, and ticket changes +on `/admin/maintenance`. Anything no trigger reaches — `report_issue`, project submission, and +category or location edits outside the editor — is left to the daily cron's backstop. The Notion +client is raw `fetch`, throttled to 3 requests a second and honouring `Retry-After`; three 5xx +answers in a row stop the push as `notion_unavailable`. Migration `0007_notion_mirror.sql` was +read before it was accepted, and its `updated_at` trigger is hand-appended. + +**Per open question 4 (previous amendment), the mirror carries reporter and author names and +emails.** §8's "Emails never enter ... the Notion mirror" and §10's "asserting emails are never +present" no longer hold for the mirror: the property-builder tests now assert that emails *are* +carried. Emails still never enter a model prompt or a log line, and `scrubSecrets` removes them, +along with tokens, from anything recorded as an error. `/admin/mirror` carries a privacy note +saying that the connected workspace will hold personal data. + +**How §4.12 differs as built.** + +- `token_ciphertext` is **nullable**, and null means disconnected. Disconnect forgets the token + but keeps the mapping and `mirror_pages`, so reconnecting creates no duplicate pages. +- `last_error` is **jsonb** `{code, entities, failed, detail}`, so the page can translate it. + `detail` is generic English and never Notion's own message. +- `notion_mirrors` adds six columns: + - `parent_page_title`; + - `push_requested_at`, the coalescing claim; + - `sync_requested_at`, which holds Sync now's 15-minute limit in the database; + - `last_run_at`, the last finish, whatever its result; + - `mapping_generation`; + - `created_at`. +- `mirror_pages` adds `source_updated_at`, a per-page revision, so a push cut short by its budget + resumes where it stopped instead of pushing everything again. +- The nightly backup blanks `token_ciphertext`. + +**Decisions.** + +- **`last_synced_at` is the claim time minus five minutes, computed in SQL**, so a transaction + that commits late is not missed. The status panel says "Every change made before this time is + in Notion." +- **A mapping change fences off a running push.** `mapping_generation` is bumped whenever the + mapping changes or an entity is reset. A push advances `last_synced_at`, and records pages, only + while the generation it claimed is still current. Otherwise it stops as superseded, and the + workflow runs another round from the new mapping. +- **Resetting an entity also marks the pages that link to it as not mirrored.** When a Tools + database is recreated (§5.8), every unit, resource, ticket and project page is pushed again, so + their Tool relations point at the new pages. +- **The 45 s budget limits when requests start, not how long they run.** A request that has + started runs until Notion answers or a fixed 30 s timeout, which is reported as `unavailable`. + Cutting a POST short after Notion has received it would create a duplicate page. +- **A mirror pushes only while its owner holds `mirror.manage` and is not banned.** This extends + §8's owner-only rule, so a demoted admin's workspace stops receiving personal data. The role + list is a constant, because step code cannot load Better Auth, and a test pins it to `can()`. +- **Sync now is refused while a push is running** (`sync_running`), without spending the owner's + 15-minute window. A workflow round that finds the mirror busy waits 30 s and tries again, up to + 10 times. +- **Unpublished projects, and hard-deleted units, resources and similar rows, have their pages + archived.** Select values are the stored machine ids. Database and property names are fixed + English in code, not next-intl strings. +- **`NOTION_API_BASE_URL` is a test-only override.** E2E 8 points it at a stub on port 3102, which + serves the same stateful fake of Notion the unit tests use. It runs as its own Playwright + project, after `intake`. +- **`useRefreshNudge` works around a stall; the stall itself is not fixed.** In the production + build (Next 16.1, React 19.2, `cacheComponents`), the page a server action refreshed was + rendered but not shown until something else updated the page. The mirror islands therefore + re-render every 200 ms for 4 s after an action. E2E 8 is the check that the workaround can be + removed after an upgrade. + +**Known gaps, not fixed.** + +- **Nothing here has run against the real Notion API.** Two behaviours in particular are + unconfirmed: a PATCH that sends `archived: false` together with properties, and select options + being created on the fly. Isaac's first connection is the check. +- **A 30 s timeout on a POST that Notion did process can still create a duplicate page.** There is + no lookup by app id to reconcile it. +- **Some changes bump no `updated_at`.** A project's tool list and attachment rows are examples. + They reach the mirror only when their owner row next changes. +- **The owner check reads the user row, not `AUTH_SUPER_ADMIN_EMAILS`.** A floor address whose row + was demoted stops pushing until `reconcileSuperAdminFloor` repairs the row. +- **A super admin cannot disconnect another admin's mirror.** A demoted owner's token stays in the + database until the account is deleted, although it no longer pushes. + +**Still to do by hand (§4.14).** Isaac creates an internal integration and a page named +"MakerLab Tools — mirror" in his Notion, shares the page with the integration, and pastes the +token and page URL into `/admin/mirror`. + +**Status.** Accepted. diff --git a/v5/.gitignore b/v5/.gitignore index c9c7848..9de57e2 100644 --- a/v5/.gitignore +++ b/v5/.gitignore @@ -19,3 +19,7 @@ tsconfig.tsbuildinfo .workflow-vitest/ .swc/ /src/app/.well-known/workflow/ + +# Local Blob store: without BLOB_READ_WRITE_TOKEN, `next dev` writes uploads, +# archived manuals and backups here (src/lib/blob-local.ts). +.blob-data/ diff --git a/v5/AGENTS.md b/v5/AGENTS.md index 1aa4f5c..4c556bd 100644 --- a/v5/AGENTS.md +++ b/v5/AGENTS.md @@ -47,15 +47,17 @@ variable list. `notion-ids.ts`. Relative imports with `.ts` extensions, no `@/` alias, no `"server-only"` — `scripts/` loads them under plain Node. - **Notion is read only by the one-time import** (`npm run import:notion`). - No request path reads Notion. A one-way mirror (app → an admin's own Notion - workspace) is a later phase, not built yet. + No request path reads Notion *as data*. The one-way mirror (app → an admin's + own Notion workspace, Phase 8) writes to Notion through its own client in + `src/lib/mirror/` — see "The Notion mirror" below. - **Every student-facing write is on Postgres** as of Phase 3. A correction goes to `feedback`, a maintenance ticket to `maintenance_logs`, a project submission to `projects` + `project_tools` — see `src/lib/data/*.ts`. `src/lib/data/notion-ids.ts` (the Phase-2 page-id bridge) has no importers left and is awaiting deletion approval, as are `/api/upload-notion` and `/api/admin/backup`. -- **No request path writes Notion** as of Phase 6. Intake's chat tool, +- **No request path writes Notion** as of Phase 6, except the mirror, which + pushes from a workflow and from its own settings page. Intake's chat tool, `identify_tools`, writes `pending_tools` rows and makes the photos it claims public; `create_tool` is **MCP-only** now and writes an unpublished Postgres draft (`createToolRecord`). See "Adding equipment" below. @@ -70,9 +72,21 @@ variable list. `POST /api/projects` answers `photosSubmitted` / `photosAttached` so the form can say it on the confirmation. A form left open overnight submits ids the cron has already swept, and thanking a student for pictures nobody has is the - quiet lie Article 4 forbids. With no `BLOB_READ_WRITE_TOKEN` the route answers + quiet lie Article 4 forbids. With no Blob store the route answers 503 `{ code: "blob_not_configured" }` and both clients show a translated "photo uploads are unavailable" — never a fabricated id (Article 4). +- **Blob locally.** `blobMode()` (`src/lib/blob-mode.ts`) is the one rule: + a `BLOB_READ_WRITE_TOKEN` → Vercel Blob; no token on Vercel (`VERCEL`) or in + a production build → **none** (503 as above, never a disk fallback); no token + in local dev → **local**: `.blob-data/` (git-ignored), metadata in + `.blob-data/.meta/`, via `src/lib/blob-local.ts`. Both write paths use it — + `lib/blob.ts`'s `getBlobStore()` and step code's `createBlobUploader()` + (`import/blob-uploader.ts`, used by the manual archiver). Public files get + `/api/dev-blob/`, served by that route (404 for + private files and outside local mode); `next.config.ts` allows those URLs for + `next/image` in dev only. `BLOB_LOCAL_DISABLE=1` forces "none" — the Vitest + setup sets it, so tests opt in to local mode with a temp `cwd`. The Notion + import still requires a real token (`createVercelBlobUploader`). - **Failing toward stale, not wrong (Article 4).** `DATABASE_URL` unset serves the PGlite demo seed with `DemoDataBanner` shown. `DATABASE_URL` set but unreachable never falls back to demo or invented data — cached pages keep @@ -256,7 +270,7 @@ Phase 5 extends both. The shape it sets: boundary is what leaves every *published* tool page prerenderable under `cacheComponents` (`npm run build` is the check); a different-looking refusal would confirm the draft exists. -- **With no `BLOB_READ_WRITE_TOKEN` the panel says photos cannot be added and +- **With no Blob store (see "Blob locally") the panel says photos cannot be added and stays usable for everything else.** `POST /api/uploads` answers 503, the Photos and Resources sections show that sentence, and reordering, removal and every text field keep working — a deployment with no Blob store is still one @@ -361,13 +375,113 @@ creates a tool (Article 5). than 14 days are discarded and their photos released (`runPendingExpiry`), just before the orphan sweep deletes them from Blob. +## The Notion mirror (`notion_mirrors`, Phase 8) + +A one-way copy of the inventory into an admin's own Notion workspace (spec +§3.8, §5.8). Postgres stays the source of truth; nothing is ever read back. + +- **One mirror per admin, found from the session.** `/admin/mirror` + (`mirror.manage`) shows only the caller's own row; no server action in + `src/app/admin/mirror/actions.ts` takes a mirror id. The four setup actions + (test, connect, create databases, save mapping) also pass + `MIRROR_SETUP_TIER` (10/min). Connect and disconnect audit + `mirror.connected` / `mirror.disconnected`. +- **The token is validated by one read, then stored encrypted.** AES-256-GCM + under a key HKDF-derived from `AUTH_SECRET` with a fixed info string + (`src/lib/mirror/token-crypto.ts`). Rotating `AUTH_SECRET` makes every stored + token unreadable; the page then asks for it again. The nightly backup blanks + the ciphertext. **Never log a token, `AUTH_SECRET` or an email** — error + text goes through `scrubSecrets`, and `last_error.detail` is generic English + built in code, never Notion's message. +- **Seven databases, fixed schemas** (`database-schemas.ts`), in dependency + order: categories, locations, tools, units, resources, maintenance, + projects. **Create databases** makes the missing ones under the shared page; + pasted ids are validated against the schema before anything is saved. +- **The push** (`push.ts`, run by `mirrorPush` in + `src/workflows/mirror-push.ts`): an overlap guard (`running_since`, 15 min), + rows newer than `last_synced_at` or than their `mirror_pages.source_updated_at`, + upsert by `mirror_pages`, archive pages of archived tools, unpublished + projects and deleted rows. 3 requests/s, 429 honoured via `Retry-After`, + 45-second budget per push (then up to six rounds, 5 s apart). The budget + stops new requests only; one already started runs to Notion's answer or a + 30 s ceiling, never cut short (an aborted create would duplicate a page). + Only a clean, complete push advances `last_synced_at`; a 401 pauses the + mirror. A round skipped because another push holds the mirror waits 30 s + and tries again (up to ten times). +- **Mapping changes and running pushes.** `setMirrorMapping` (when the + mapping changes) and `resetMirrorEntities` bump `mapping_generation`; a push + records pages and advances `last_synced_at` only while the mirror is still + at the generation it claimed, and otherwise stops as `incomplete` for + another round. `resetMirrorEntities` also marks the pages of every entity + with a relation into the reset ones (`relationDependents`) as not mirrored, + so their links are rewritten to the new pages. +- **Only a current admin's mirror pushes.** Every claim except Sync now (whose + server action already checked `mirror.manage`) requires the owner's `user` + row to hold a role in `MIRROR_OWNER_ROLES` and not be banned; demoting or + banning an admin stops their mirror. Sync now is refused (`sync_running`) + while another push holds the mirror, without spending the 15 minutes. +- **What it carries.** Every tool (with a Published checkbox), units, + resources, categories, locations, maintenance logs and published projects; + public attachments as external files, never private ones. **Reporter, + assignee and author names and emails are carried** (open question 4, + answered 2026-09-23) — the mirror's workspace holds personal data. Emails + still never enter a model prompt or a log line. +- **Three triggers.** (1) `requestMirrorPush()` (`src/lib/mirror/trigger.ts`) + after a committed write — approving a tool, every tool-editor write + (`tool-write-context.ts`), publishing a project, working a maintenance + ticket. It never throws, costs one query when nobody has a mirror, and + starts `mirrorPushAfterChange`, which sleeps two minutes so a burst + coalesces. (2) **Sync now**, once per mirror per 15 minutes. (3) The daily + cron's `mirror` stage (`src/lib/cron/mirror-backstop.ts`), for any mirror + whose data is newer than its last sync. +- **Notion is called with raw `fetch`** (`notion-client.ts`, API version + `2022-06-28`) — no SDK. `NOTION_API_BASE_URL` overrides the base URL for the + E2E stub only; production never sets it. + +## Archived manuals (link rot) + +A manufacturer moves a PDF and the tool loses its manual. So each manual link +is copied into Blob once, and the tool page and the chat prefer the copy. + +- **No table of its own.** The copy is an `attachments` row owned by the + resource — public, `application/pdf`, `source_key = + manual::` (`src/lib/data/manual-archives.ts`). The + resource keeps its own `url`, the manufacturer's link. A copy whose key does + not match the resource's current `url` is stale: hidden everywhere, and + released to the orphan sweep when the new link is archived. +- **`archiveManual(resourceId)`** (`src/lib/manuals/archive.ts`) archives a + Manual, or any resource whose link answers a PDF. The body must *be* a PDF + (`%PDF-`, or `application/pdf` that is not markup) — an HTML product page is + refused. 30 s, 25 MB. Skips a resource that already holds a PDF (uploaded, + imported, or this link's copy), and skips as `blob_not_configured` without a + token. It returns `archived | skipped | failed` with a reason and never + throws for an expected failure; it logs the link's host only, never the path + or query. Step code: it writes Blob through `import/blob-uploader.ts`, not + the `server-only` `lib/blob.ts`. +- **Runs in a workflow**, `archiveManuals(resourceIds)` + (`src/workflows/archive-manuals.ts`), one step per resource, `maxRetries = 2`, + retrying only a transient failure (network, 5xx/429, a Blob write, the + database). Started by `requestManualArchive()` (`src/lib/manuals/trigger.ts`) + after approving a tool, after MCP `create_tool`, and after the editor adds a + resource or changes its link or type. It never throws and never fails the + write that called it. +- **The daily cron's `manuals` stage** (`src/lib/cron/manual-archive.ts`) hands + up to ten due Manuals to one run each night — the backfill for imported + manuals and the backstop for a start that never happened. The window moves + each night and wraps, so a manual that always fails cannot stall it. +- **Readers.** `resourceLinks` gives an archived manual **one** link, to the + copy, with the manufacturer's URL as `sourceHref` (the tool page shows only + `href`). `listResourcesForTool` sets it apart as `archivedUrl` and leaves it + out of `fileUrls`; the chat attaches `archivedUrl` first, so one manual is + attached once, and "(attached)" matches the copy or the source. + ## Key files | Path | Purpose | |---|---| | `src/lib/site-config.ts` | White-label branding (env-driven, all have defaults) | | `src/lib/db/client.ts` | `getDb()`, `dataSubstrate()`, `pingDb()` — the one entry point to Postgres/PGlite | -| `src/lib/notion.ts` | Notion API client — used by the one-time import and its scripts (and the retired `/api/admin/backup`, awaiting deletion); no request path reads or writes Notion | +| `src/lib/notion.ts` | Notion API client — used by the one-time import and its scripts (and the retired `/api/admin/backup`, awaiting deletion); no request path reads or writes Notion through it (the mirror has its own client) | | `src/lib/data/attachments.ts` | `attachments` rows: create, claim onto an owner, reorder, release, list orphans, delete | | `src/lib/data/revision.ts` | The editor's concurrency token — `extract(epoch from updated_at)::text`, **never a `Date`** (read the docstring before touching a conflict check) | | `src/lib/data/tools.ts` / `units.ts` | Row-level inventory writes, every one revision-checked. Tools are archived, never deleted | @@ -414,7 +528,15 @@ creates a tool (Article 5). | `src/app/api/chat/route.ts` | Claude chat: streaming, capability tools (`get_unit_details`, `report_issue`, `identify_tools`, …) plus `web_search` / `web_fetch`, PDF manual attach | | `src/app/api/mcp/route.ts` | MCP JSON-RPC server (5 tools), bearer-token auth | | `src/app/api/uploads/route.ts` | The one upload route → Vercel Blob + an `attachments` row | -| `src/app/api/cron/daily/route.ts` | The single nightly cron (`vercel.json`): backup, then pending-item expiry, then orphaned-upload cleanup | +| `src/app/api/cron/daily/route.ts` | The single nightly cron (`vercel.json`): backup, then pending-item expiry, then orphaned-upload cleanup, then the mirror backstop, then the manual archive backfill | +| `src/lib/manuals/*` | The manual archive: `archive` (`archiveManual`), `steps`, `start` (the one `workflow/api` import), `trigger` (`requestManualArchive`, never throws) | +| `src/workflows/archive-manuals.ts` | `archiveManuals(resourceIds)` — one step per resource | +| `src/lib/data/manual-archives.ts` / `src/lib/cron/manual-archive.ts` | The archive's key, stale-copy release and the nightly due list; the cron stage | +| `src/lib/db/schema/mirror.ts`, `src/lib/data/mirrors.ts` / `mirror-pages.ts` | `notion_mirrors` and `mirror_pages`; every claim (run, Sync now, coalesced push) is one conditional `UPDATE` | +| `src/lib/mirror/*` | The mirror: `notion-client` (raw fetch, throttle, 429), `token-crypto`, `credentials`, `notion-id`, `database-schemas`, `databases` (create / validate pasted ids), `source` (what changed), `properties` (pure row → Notion builders), `push`, `steps`, `start`, `trigger`, `connect` | +| `src/workflows/mirror-push.ts` | `mirrorPush(mirrorId)` and `mirrorPushAfterChange()` — the `"use workflow"` functions | +| `src/app/admin/mirror/` + `src/components/admin/Mirror*.tsx` | The settings page, its seven server actions, and the four islands (`MirrorConnect`, `MirrorMapping`, `MirrorStatus`, `MirrorControls`) | +| `test/fakes/notion-fake.ts` | The in-memory Notion every mirror test (and the E2E stub) talks to | | `src/app/api/admin/revalidate/route.ts` | Cache invalidation (`tools.edit`, or `x-admin-secret` for session-less callers) | | `src/components/ChatFab.tsx` | Chat UI (`useChat`, citations stripped, photo upload) | | `src/app/page.tsx`, `tools/[id]/page.tsx` | Gallery + tool detail | @@ -452,13 +574,17 @@ first), `npm run test:coverage`. - **The whole suite runs with every environment variable unset.** Reads *and* writes go to an in-process PGlite database seeded with demo data; Vercel Blob is stubbed at the `src/lib/blob.ts` seam (`vi.mock`), never called for real. - No write reaches Notion, so no test stubs it for one. + `vitest.setup.ts` sets `BLOB_LOCAL_DISABLE=1`, so "no token" still means "no + store"; the local-store tests opt in and write to a temp folder. + Only the mirror writes Notion; its tests talk to an in-memory Notion + (`test/fakes/notion-fake.ts`) through MSW, never to `api.notion.com`. - **Two Vitest projects.** `unit` is the existing config; `workflow` (`vitest.workflow.config.ts`) runs `*.workflow.test.ts` under `@workflow/vitest`, where the model is stubbed with MSW on `api.anthropic.com` because `vi.mock` does not reach step code. - E2E boots its own server on **port 3100** with `DATABASE_URL` unset (PGlite demo catalog) and intercepts `/api/chat` — it never touches your `:3000` dev server or real services. - **The intake E2E is the exception** (`e2e/intake.spec.ts`): it needs `identify_tools` and the research workflow to run server-side, so the model is stubbed at the provider boundary by a second local server (`e2e/stubs/anthropic-stub.ts`, reached through `ANTHROPIC_BASE_URL`), and the workflow runs on the SDK's local world. It is its own Playwright project that runs after every other spec, because approving publishes a third tool into the shared demo database. +- **The mirror E2E** (`e2e/mirror.spec.ts`) is the same shape: Notion is a local stub (`e2e/stubs/notion-stub.ts`, port 3102, reached through `NOTION_API_BASE_URL`) serving the same fake, and the project runs after `intake`, last of all. - Tests are colocated (`*.test.ts(x)` next to source); shared harness in `test/`. - **Read these before writing tests:** `TESTING.md` (runbook), `test/README.md` (harness internals + the `streamText`-capture and env-stubbing patterns), and `docs/specs/2026-05-29-v5-test-suite-design.md` (design + coverage matrix). The harness deps/scripts are already wired — don't hand-edit `package.json` for them. @@ -475,5 +601,13 @@ npm run test:all # full test suite ## Gotchas - Because `cacheComponents` is enabled, API routes **cannot set `runtime`** — they use the default Node runtime. +- **A server action's re-render can sit uncommitted.** In the production build + (Next 16.1, React 19.2, `cacheComponents`), the page a server action + refreshed with `revalidatePath` finished rendering and was not shown until + something else updated the page — and a later `router.refresh()` queued + behind it. Islands that depend on the re-rendered page (the mirror page's + four) call `useRefreshNudge()` (`src/components/admin/use-refresh-nudge.ts`) + after a successful action; islands that keep their own confirmed state (the + queues, `RoleSelect`) are unaffected. E2E scenario 8 is the canary. - The in-memory rate limiter is a per-process singleton; it resets on cold start (fine for abuse prevention). Upstash backs it only when **both** `UPSTASH_REDIS_REST_*` vars are set. - Python scripts under `scripts/` use Node with `--experimental-strip-types`; they are migration/maintenance tools, not part of the app build. diff --git a/v5/TESTING.md b/v5/TESTING.md index c7ec09a..0e84447 100644 --- a/v5/TESTING.md +++ b/v5/TESTING.md @@ -52,6 +52,17 @@ Playwright boots its own dev server (see E2E notes below), so no separate default handlers in `test/msw/handlers.ts`. Lifecycle (start / reset / stop) is managed in `vitest.setup.ts`. **Unhandled outbound requests fail the test by design** (`onUnhandledRequest: "error"`). +- **The Notion mirror** (`src/lib/mirror/*`) is tested against a stateful + in-memory Notion, `test/fakes/notion-fake.ts` (`createNotionFake`), which + checks the bearer token and the `Notion-Version` header, validates page + properties against the database schema, and can be told to fail + (`failNext`, including 429 with `Retry-After`). Install it into a test's MSW + server with `useNotionFake(server, fake)` from `test/msw/notion-mirror.ts` — + imported under another name (`import { useNotionFake as installNotionFake }`) + outside a component, because ESLint's hooks rule reads any `use*` call as a + hook. Workflow tests use the same fake through MSW, since `vi.mock` does not + reach step code. PGlite's session time zone is the machine's, so compare + `timestamptz` text in SQL (`$1::timestamptz = …`), never as strings. - **`vi.mock("next/cache", …)`** — `catalog.ts` uses `cacheTag`/`cacheLife` and `admin/revalidate/route.ts` uses `revalidateTag`; these only work inside a Next build. Mock them with the `nextCacheMock()` factory from @@ -142,6 +153,12 @@ test (the setup file). The in-memory rate limiter is a per-process singleton `Map` — use distinct keys per test, or `resetModules()` + re-import for a fresh window. +**Blob mode.** The setup file sets `BLOB_LOCAL_DISABLE=1`, so with no +`BLOB_READ_WRITE_TOKEN` a test sees "no store" (`blob_not_configured`), as a +deploy without one does. To test the local `.blob-data/` store, stub +`BLOB_LOCAL_DISABLE` / `VERCEL` to `""` and `NODE_ENV` to `"development"`, and +point `process.cwd()` at a temp folder (`src/lib/blob-local.test.ts`). + **streamText-capture pattern (chat route).** The chat route's tool `execute` functions are inline and its helpers are module-private, so don't unit-test them directly. Instead mock `ai`'s `streamText` (spreading `...actual`) to capture @@ -167,6 +184,17 @@ directly. Also mock `@ai-sdk/anthropic`. The full verified snippet is in (`intake`) that depends on `chromium`, so it runs after every other spec: approving publishes a tool into the demo database they all share. Run it alone with `npx playwright test --project=intake --no-deps`. +- **And `e2e/mirror.spec.ts`** (§10 scenario 8), the same shape for Notion: the + mirror calls Notion from server actions and workflow steps, so a third web + server, `e2e/stubs/notion-stub.ts` on port 3102, answers `/v1/*` with the + in-memory fake the Vitest suites use (`test/fakes/notion-fake.ts`), and the + app reaches it through `NOTION_API_BASE_URL` (test-only; production never + sets it). Fixture values (fake token, page id and URL) are in + `e2e/stubs/notion-fixture.ts`. It is its own project (`mirror`) that depends + on `intake`, so it runs last of all: while a mirror is connected every write + in the app schedules a push, and no other spec may be writing then. One + test, `retries: 0` — each step is the next one's precondition. Run it alone + with `npx playwright test --project=mirror --no-deps`. - `reuseExistingServer: false` — Playwright **always** boots its own fresh PGlite-backed server on the dedicated port 3100. This means E2E never collides with (or accidentally reuses) a `next dev` you have running on the diff --git a/v5/e2e/mirror.spec.ts b/v5/e2e/mirror.spec.ts new file mode 100644 index 0000000..dce8f47 --- /dev/null +++ b/v5/e2e/mirror.spec.ts @@ -0,0 +1,113 @@ +import { test, expect, type Locator } from "@playwright/test"; + +import { DEMO_ACCOUNTS } from "../src/lib/db/demo-seed"; +import { NOTION_STUB_PAGE_TITLE, NOTION_STUB_PAGE_URL, NOTION_STUB_TOKEN } from "./stubs/notion-fixture"; +import { signIn } from "./utils/session"; + +/** + * The Notion mirror, end to end (data platform design spec §3.8, §10 E2E + * scenario 8): an admin connects a mirror, presses Sync now, and sees a + * last-synced time. + * + * **Notion is stubbed at the API boundary, not in the browser.** The mirror + * reads Notion from server actions and writes it from workflow steps, neither + * of which a `page.route()` can reach. So the server talks to + * `e2e/stubs/notion-stub.ts` on localhost — the same in-memory Notion the + * Vitest suites install through MSW — and everything in between is the real + * app: the server actions and their gate, the token encryption, Create + * databases, the `mirrorPush` workflow on the SDK's local world, and the + * status panel polling until the push lands. + * + * **It is the director's mirror** — Isaac's, the first one (§3.8 "Owners") — + * and it runs in its own Playwright project after the parallel specs, because + * a connected mirror turns every later change in the app into a scheduled + * push. It disconnects at the end for the same reason. + * + * **One test, not retried.** Each step is the next one's precondition, and a + * second attempt would meet the first attempt's mirror — connected, mapped, + * inside its 15-minute Sync now window. + */ + +test.describe.configure({ retries: 0 }); + +/** The value beside a `
` label in the status panel. */ +function fact(panel: Locator, label: string): Locator { + return panel.locator("dt", { hasText: label }).locator("xpath=following-sibling::dd[1]"); +} + +test("an admin connects a mirror, creates its databases, syncs, and sees when it last synced", async ({ + page, + context, + baseURL, +}) => { + // Create databases makes seven, and the push writes every demo row, all at + // three requests a second. + test.setTimeout(180_000); + await signIn(context, DEMO_ACCOUNTS.superAdmin, baseURL); + + // ── Step 1: the page, reached from /admin ───────────────────────────────── + await page.goto("/admin"); + const surfaces = page.getByRole("list", { name: "Admin pages your account can open" }); + await surfaces.getByRole("link", { name: "Notion mirror" }).click({ timeout: 15_000 }); + await expect(page).toHaveURL(/\/admin\/mirror$/); + await expect(page.getByRole("heading", { name: "Notion mirror", level: 2 })).toBeVisible(); + await expect(page.getByRole("heading", { name: "Before you connect" })).toBeVisible(); + + // ── Step 2: test the connection, then connect ───────────────────────────── + const token = page.getByLabel("Integration token"); + await expect(token).toHaveAttribute("type", "password"); + await token.fill(NOTION_STUB_TOKEN); + await page.getByLabel("Page URL").fill(NOTION_STUB_PAGE_URL); + + await page.getByRole("button", { name: "Test connection" }).click(); + await expect(page.getByText(`Found the page “${NOTION_STUB_PAGE_TITLE}”.`)).toBeVisible({ timeout: 15_000 }); + + await page.getByRole("button", { name: "Connect", exact: true }).click(); + + // The page re-renders into its connected state: status, controls, mapping. + const status = page.getByRole("region", { name: "Status" }); + await expect(status.getByText(`Connected to “${NOTION_STUB_PAGE_TITLE}”.`)).toBeVisible({ timeout: 15_000 }); + await expect(fact(status, "Last synced")).toHaveText("Never"); + // The token went in and never came back out. + await expect(page.locator("body")).not.toContainText(NOTION_STUB_TOKEN); + + // ── Step 3: Create databases → seven ids ────────────────────────────────── + const mapping = page.getByRole("table", { name: "Notion databases, one per table" }); + await expect(mapping.getByText("Not set")).toHaveCount(7); + await page.getByRole("button", { name: "Create databases" }).click(); + await expect(page.getByText("Created 7 databases.")).toBeVisible({ timeout: 30_000 }); + await expect(mapping.getByText("Not set")).toHaveCount(0, { timeout: 15_000 }); + await expect(mapping.locator("code")).toHaveCount(7); + for (const id of await mapping.locator("code").allTextContents()) { + expect(id).toMatch(/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/); + } + + // ── Step 4: Sync now → the workflow pushes → a last-synced time and OK ──── + const syncNow = page.getByRole("button", { name: "Sync now" }); + await expect(syncNow).toBeEnabled(); + await syncNow.click(); + await expect(page.getByText("Sync started.")).toBeVisible({ timeout: 15_000 }); + + // The panel polls every five seconds while the push is requested or running. + await expect(fact(status, "Last result")).toHaveText("OK", { timeout: 90_000 }); + const lastSynced = fact(status, "Last synced").locator("time"); + await expect(lastSynced).toBeVisible(); + await expect(lastSynced).toHaveAttribute("datetime", /^\d{4}-\d{2}-\d{2}T/); + await expect(status.getByText("Last error")).toHaveCount(0); + + // ── Step 5: a second Sync now is refused, with the 15-minute reason ─────── + await expect(syncNow).toBeDisabled(); + await expect(page.getByText(/Sync now runs once every 15 minutes\./)).toBeVisible(); + // Minutes, not a raw timestamp; how many depends on how long the push took. + await expect(page.getByText(/Available again in \d+ minutes?\./)).toBeVisible(); + + // ── Step 6: disconnect, so nothing later pushes to the stub ─────────────── + await page.getByRole("button", { name: "Disconnect" }).click(); + await page.getByRole("button", { name: "Yes, disconnect" }).click(); + await expect(page.getByRole("heading", { name: "The connection needs a new token" })).toBeVisible({ + timeout: 15_000, + }); + // The mapping is kept, so reconnecting would update the same pages. + await expect(mapping.locator("code")).toHaveCount(7); + await expect(page.getByLabel("Integration token")).toHaveValue(""); +}); diff --git a/v5/e2e/stubs/notion-fixture.ts b/v5/e2e/stubs/notion-fixture.ts new file mode 100644 index 0000000..eb9937e --- /dev/null +++ b/v5/e2e/stubs/notion-fixture.ts @@ -0,0 +1,31 @@ +/** + * The Notion workspace the mirror E2E connects to (data platform spec §10, E2E + * scenario 8). + * + * Shared by the spec, by `notion-stub.ts` and by `playwright.config.ts`, so + * the token the test types is the token the stub accepts and the page it + * pastes is the page the stub has. No imports: the stub runs under plain Node. + * + * **Nothing here is a real credential.** The token is shaped like one so the + * app's validation accepts it, and it is accepted by nothing but the stub on + * this machine. + */ + +/** The port the stub listens on. The app reaches it through `NOTION_API_BASE_URL`. */ +export const NOTION_STUB_PORT = 3102; + +export const NOTION_STUB_ORIGIN = `http://localhost:${NOTION_STUB_PORT}`; + +/** An internal-integration token as Notion shapes them. Fake. */ +export const NOTION_STUB_TOKEN = "ntn_e2eStubTokenNotReal0000000000"; + +/** The page the admin "shared with the integration" (§4.14). */ +export const NOTION_STUB_PAGE_ID = "5e2e0c7a-3b1d-4f6a-9c8e-0d1f2a3b4c5d"; + +export const NOTION_STUB_PAGE_TITLE = "MakerLab Tools — mirror"; + +/** The page as an admin pastes it: the browser's address bar, slug and all. */ +export const NOTION_STUB_PAGE_URL = `https://www.notion.so/makerlab/MakerLab-Tools-mirror-${NOTION_STUB_PAGE_ID.replace( + /-/g, + "" +)}?pvs=4`; diff --git a/v5/e2e/stubs/notion-stub.ts b/v5/e2e/stubs/notion-stub.ts new file mode 100644 index 0000000..9bd0daa --- /dev/null +++ b/v5/e2e/stubs/notion-stub.ts @@ -0,0 +1,81 @@ +import { createServer, type IncomingMessage } from "node:http"; + +import { createNotionFake } from "../../test/fakes/notion-fake.ts"; +import { + NOTION_STUB_ORIGIN, + NOTION_STUB_PAGE_ID, + NOTION_STUB_PAGE_TITLE, + NOTION_STUB_PORT, + NOTION_STUB_TOKEN, +} from "./notion-fixture.ts"; + +/** + * A stand-in for the Notion API, for the mirror E2E only (data platform spec + * §10, E2E scenario 8). + * + * The mirror calls Notion from server actions and from workflow steps, which + * no browser-side intercept can reach. So the Playwright server boots with + * `NOTION_API_BASE_URL` pointing here, and every `/v1/*` request is answered by + * `createNotionFake` — the same in-memory Notion the Vitest suites install + * through MSW, so a push clicked through in a browser meets the Notion the + * integration tests met. Seeded with one page, the one the admin shares with + * the integration (§4.14). + * + * Run with `node --experimental-strip-types`: the fake is written to load + * that way (no imports, no enums, no parameter properties). It logs one line + * per request with the method, path and status — never a header, so never the + * token, and never a body. + */ + +const fake = createNotionFake({ + token: NOTION_STUB_TOKEN, + pages: [{ id: NOTION_STUB_PAGE_ID, title: NOTION_STUB_PAGE_TITLE }], +}); + +function readBody(req: IncomingMessage): Promise { + return new Promise((resolve, reject) => { + const chunks: Buffer[] = []; + req.on("data", (chunk: Buffer) => chunks.push(chunk)); + req.on("end", () => resolve(Buffer.concat(chunks).toString("utf8"))); + req.on("error", reject); + }); +} + +/** Node's header record, flattened into the plain record the fake reads. */ +function headersOf(req: IncomingMessage): Record { + const out: Record = {}; + for (const [name, value] of Object.entries(req.headers)) { + if (value === undefined) continue; + out[name] = Array.isArray(value) ? value.join(", ") : value; + } + return out; +} + +const server = createServer(async (req, res) => { + const url = new URL(req.url ?? "/", NOTION_STUB_ORIGIN); + + // Playwright's readiness probe. + if (url.pathname === "/") { + res.writeHead(200, { "content-type": "text/plain" }); + res.end("notion stub"); + return; + } + + if (!url.pathname.startsWith("/v1/")) { + res.writeHead(404, { "content-type": "text/plain" }); + res.end("not found"); + return; + } + + const method = req.method ?? "GET"; + const body = method === "GET" || method === "HEAD" ? "" : await readBody(req); + const answer = fake.handle(method, url.pathname, headersOf(req), body); + + res.writeHead(answer.status, answer.headers); + res.end(answer.body === undefined ? "" : JSON.stringify(answer.body)); + console.info(`[notion-stub] ${method} ${url.pathname} → ${answer.status}`); +}); + +server.listen(NOTION_STUB_PORT, () => { + console.info(`[notion-stub] listening on ${NOTION_STUB_ORIGIN}`); +}); diff --git a/v5/messages/en.json b/v5/messages/en.json index 9eafa8e..54daaef 100644 --- a/v5/messages/en.json +++ b/v5/messages/en.json @@ -253,6 +253,9 @@ "lede": "Check each row, untick anything you don’t want researched yet, then press Research.", "selectAllAria": "Select all rows", "selectRowAria": "Select {name}", + "selectAllVisible": "Research all", + "selectRowVisible": "Research this one", + "selectRowBlocked": "Choose what to do with the duplicate first", "columnSelect": "Select", "columnPhoto": "Photo", "columnName": "Name", @@ -367,7 +370,6 @@ "indexTitle": "Admin surfaces", "indexListLabel": "Admin pages your account can open", "indexNothingYet": "Nothing here is open to your account yet.", - "indexMoreComing": "The Notion mirror arrives in a later phase.", "inventoryTitle": "Inventory", "inventoryLede": "Every tool the lab owns — drafts and archived records included. Needs attention is where a review starts.", "maintenanceTitle": "Maintenance", @@ -380,6 +382,8 @@ "usersLede": "Everyone who has signed in. A role change takes effect on that person's next request.", "intakeTitle": "Intake", "intakeLede": "New equipment, researched in the background and waiting for approval.", + "mirrorTitle": "Notion mirror", + "mirrorLede": "Your own copy of the inventory in a Notion workspace, pushed after every change, and when it last was.", "tableLabel": "People and their roles", "columnPerson": "Person", "columnRole": "Role", @@ -788,6 +792,148 @@ "start_failed": "Research could not start. The item is still queued — press Retry.", "failed": "That did not go through. Nothing was changed; try again." } + }, + "mirror": { + "unavailable": "The mirror's settings could not be read. The database may be unreachable — try again in a moment.", + "keyUnavailableTitle": "This deployment cannot store a Notion token", + "keyUnavailableBody": "A token is encrypted with a key derived from AUTH_SECRET before it is stored, and AUTH_SECRET is not set here. Set it and redeploy, then come back.", + "howToTitle": "Before you connect", + "howToIntegration": "In your own Notion workspace, create an internal integration (Settings → Connections → Develop or manage integrations) and copy its token.", + "howToPage": "Create a page called “MakerLab Tools — mirror”.", + "howToShare": "Share that page with the integration: ••• → Connections → add it.", + "howToPaste": "Paste the token and the page's URL below and connect. Create databases does the rest.", + "privacyNote": "The mirror copies every tool, unit, resource, maintenance ticket and published project into that workspace — including the names and email addresses of the people who reported tickets and wrote projects. Share the page accordingly.", + "needsTokenTitle": "The connection needs a new token", + "needsTokenDisconnected": "The token was forgotten when the mirror was disconnected. Connect again to carry on — the databases and pages below are kept, so nothing is duplicated.", + "needsTokenRejected": "Notion no longer accepts the stored token: it was revoked, or this deployment's AUTH_SECRET changed. Connect again with a new token — the databases and pages below are kept, so nothing is duplicated.", + "connectedTo": "Connected to “{title}”.", + "connectedToUntitled": "Connected to an untitled page.", + "disconnectedFrom": "Disconnected. It last pushed to “{title}”.", + "disconnectedFromUntitled": "Disconnected. It last pushed to an untitled page.", + "entities": { + "categories": "Categories", + "locations": "Locations", + "tools": "Tools", + "units": "Units", + "resources": "Resources", + "maintenance": "Maintenance", + "projects": "Projects" + }, + "connect": { + "title": "Connect", + "tokenLabel": "Integration token", + "tokenHint": "Starts with ntn_ or secret_. It is encrypted before it is stored, and never shown again.", + "pageLabel": "Page URL", + "pageHint": "The address of the page you shared with the integration.", + "test": "Test connection", + "testing": "Testing…", + "connect": "Connect", + "connecting": "Connecting…", + "found": "Found the page “{title}”.", + "foundUntitled": "Found the page. It has no title.", + "connected": "Connected to “{title}”.", + "connectedUntitled": "Connected." + }, + "mapping": { + "title": "Databases", + "tableLabel": "Notion databases, one per table", + "columnTable": "Table", + "columnDatabase": "Notion database", + "notSet": "Not set", + "create": "Create databases", + "creating": "Creating…", + "createHint": "Makes each missing database under the connected page, with the properties the push writes. Databases that already exist are kept.", + "created": "Created {count, plural, one {# database} other {# databases}}.", + "allExisted": "Every database already exists. Nothing was created.", + "createdBeforeStopping": "Created {count, plural, one {# database} other {# databases}} before stopping: {names}. They are in the mapping.", + "stoppedAt": "It stopped at {entity}.", + "readOnly": "Connect again to change the databases. The mapping is kept as it was.", + "pasteTitle": "Use databases you already have", + "pasteHint": "Paste a database's URL or id. Each one is checked against the properties the push writes, and nothing is saved unless every one matches. A table left blank keeps its database.", + "pasteLabel": "{entity} database", + "save": "Save mapping", + "saving": "Checking…", + "saved": "Saved. Every pasted database matches.", + "problems": { + "invalid_database_id": "Not a Notion database URL or id.", + "database_not_found": "Not found. Share the database with the integration, or check the id.", + "schema_mismatch": "Its properties do not match what the push writes." + }, + "missing": "Missing: {names}", + "wrongType": "Wrong type: {names}" + }, + "status": { + "title": "Status", + "lastSynced": "Last synced", + "lastSyncedHint": "Every change made before this time is in Notion.", + "lastRun": "Last run", + "lastResult": "Last result", + "never": "Never", + "noResult": "No push yet", + "result": { + "ok": "OK", + "partial": "Partial", + "failed": "Failed" + }, + "lastError": "Last error", + "errorEntities": "Tables: {names}", + "errorFailed": "{count, plural, one {# row} other {# rows}} failed.", + "errorDetail": "What Notion said", + "paused": "Paused. Nothing is pushed until you resume.", + "running": "Pushing to Notion now…", + "syncPending": "Sync requested — the push starts in a few seconds.", + "scheduled": "Recent changes will be pushed in a couple of minutes.", + "neverSynced": "Nothing has been pushed yet. Create the databases, then press Sync now." + }, + "lastError": { + "unauthorized": "Notion refused the token, so the mirror paused itself. The token was revoked or is wrong — connect again with a new one.", + "token_unreadable": "The stored token cannot be decrypted: this deployment's AUTH_SECRET changed. Connect again with the token.", + "key_unavailable": "AUTH_SECRET is not set, so the stored token cannot be read.", + "database_not_found": "A mirror database was not found — it may have been deleted in Notion. Create databases recreates only the missing ones.", + "schema_mismatch": "A mirror database's properties no longer match what the push writes. Somebody may have edited them in Notion.", + "rows_failed": "Some rows could not be pushed. They are tried again on the next push.", + "budget_exhausted": "The push ran out of time before it finished. The rest goes on the next push.", + "notion_unavailable": "Notion did not answer. The push is tried again on the next trigger.", + "unknown": "The push failed for a reason the app does not recognise. It is tried again on the next trigger." + }, + "controls": { + "label": "Mirror controls", + "syncNow": "Sync now", + "syncing": "Starting…", + "syncStarted": "Sync started.", + "syncLimit": "Sync now runs once every 15 minutes.", + "syncAvailableIn": "Available again in {minutes, plural, one {# minute} other {# minutes}}.", + "syncNeedsResume": "Resume the mirror to sync it.", + "syncNeedsMapping": "Create or map the databases first.", + "syncRunning": "A push is running now.", + "pause": "Pause", + "resume": "Resume", + "pausedDone": "Paused.", + "resumedDone": "Resumed.", + "disconnect": "Disconnect", + "disconnectConfirm": "Forget the token? The databases and pages stay in Notion and in the mapping, so connecting again picks up where this left off.", + "disconnectYes": "Yes, disconnect", + "disconnectNo": "Cancel", + "working": "Working…" + }, + "errors": { + "invalid_token": "That does not look like a Notion integration token. Copy it again from the integration's page.", + "invalid_page": "That is not a Notion page URL or id.", + "invalid_database_id": "Paste at least one Notion database URL or id, and check the ones marked below.", + "unauthorized": "Notion refused this token. Check that it is the integration's current token.", + "page_not_found": "Notion could not find that page. Share it with the integration (••• → Connections), then try again.", + "database_not_found": "A database could not be found. Share it with the integration, or check the id.", + "schema_mismatch": "A database's properties do not match what the mirror writes. Nothing was saved.", + "notion_unavailable": "Notion did not answer. Nothing was changed — try again in a minute.", + "key_unavailable": "This deployment cannot store a Notion token: AUTH_SECRET is not set.", + "token_unreadable": "The stored token cannot be read any more. Connect again with the token.", + "not_connected": "No mirror is connected. Connect one first.", + "not_mapped": "No databases are mapped yet. Create databases first.", + "mirror_paused": "The mirror is paused. Resume it first.", + "sync_too_soon": "Sync now runs once every 15 minutes.", + "sync_running": "A push is running now. Press Sync now again once it finishes.", + "start_failed": "The push could not be started. Nothing was pushed — try again." + } } } } diff --git a/v5/next.config.ts b/v5/next.config.ts index dce000d..bd4e00b 100644 --- a/v5/next.config.ts +++ b/v5/next.config.ts @@ -4,6 +4,18 @@ import { withWorkflow } from "workflow/next"; const withNextIntl = createNextIntlPlugin("./src/i18n/request.ts"); +// Without `BLOB_READ_WRITE_TOKEN`, `next dev` stores uploads in `.blob-data/` +// and serves public ones from `/api/dev-blob/…` on its own origin +// (src/lib/blob-local.ts). Allowed for next/image in development only; a +// production build never has these patterns. +const isDev = process.env.NODE_ENV !== "production"; +const devBlobPatterns: NonNullable["remotePatterns"]> = isDev + ? [ + { protocol: "http", hostname: "localhost", pathname: "/api/dev-blob/**" }, + { protocol: "http", hostname: "127.0.0.1", pathname: "/api/dev-blob/**" }, + ] + : []; + const nextConfig: NextConfig = { cacheComponents: true, // PGlite ships its WASM build and its extension tarballs as files it locates @@ -53,7 +65,11 @@ const nextConfig: NextConfig = { // The Notion/S3/Airtable patterns above can go once every image has been // re-imported to Vercel Blob; until then, rows imported before the switch // may still reference them. + ...devBlobPatterns, ], + // The local Blob store's files are served by this same dev server, and the + // optimizer refuses loopback addresses unless told otherwise. Dev only. + dangerouslyAllowLocalIP: isDev, minimumCacheTTL: 3600, }, }; diff --git a/v5/playwright.config.ts b/v5/playwright.config.ts index a981728..713b9d5 100644 --- a/v5/playwright.config.ts +++ b/v5/playwright.config.ts @@ -1,6 +1,7 @@ import { defineConfig, devices } from "@playwright/test"; import { ANTHROPIC_STUB_ORIGIN } from "./e2e/stubs/intake-fixture"; +import { NOTION_STUB_ORIGIN } from "./e2e/stubs/notion-fixture"; // E2E runs against a PRODUCTION build (`next build && next start`) booted with // no `DATABASE_URL` and no Notion env, so the catalogue is the in-process @@ -15,6 +16,11 @@ import { ANTHROPIC_STUB_ORIGIN } from "./e2e/stubs/intake-fixture"; // answers the Messages API on localhost, and the app reaches it through // ANTHROPIC_BASE_URL. It is still no network: the stub is on this machine. // +// The mirror scenario (e2e/mirror.spec.ts) is the same shape: the mirror calls +// Notion from server actions and workflow steps, so a third web server +// (e2e/stubs/notion-stub.ts) answers the Notion API on localhost with the same +// in-memory fake the Vitest suites use, reached through NOTION_API_BASE_URL. +// // Why not `next dev`: with parallel workers, the first request to each route // compiled it on demand and Turbopack rewrote the root layout's client chunk // while another worker was downloading it (ERR_CONTENT_LENGTH_MISMATCH), so @@ -46,7 +52,7 @@ export default defineConfig({ { name: "chromium", use: { ...devices["Desktop Chrome"] }, - testIgnore: /intake\.spec\.ts/, + testIgnore: /(intake|mirror)\.spec\.ts/, }, { // Last, on purpose. Approving an item publishes a third tool into the one @@ -58,6 +64,19 @@ export default defineConfig({ testMatch: /intake\.spec\.ts/, dependencies: ["chromium"], }, + { + // Last of all: while a mirror is connected, every change in the app + // schedules a push, and no other spec should be writing then. That + // includes intake, whose approval would schedule a coalesced push that + // could meet Sync now's push at the overlap guard and leave the status + // panel waiting on a run that sleeps two minutes first. Depending on + // "intake" (which depends on "chromium") serialises all three, so the + // mirror scenario meets a quiet app. The spec disconnects at the end. + name: "mirror", + use: { ...devices["Desktop Chrome"] }, + testMatch: /mirror\.spec\.ts/, + dependencies: ["intake"], + }, ], webServer: [ { @@ -67,6 +86,13 @@ export default defineConfig({ reuseExistingServer: false, timeout: 30_000, }, + { + // The stand-in for the Notion API, for the mirror scenario. + command: "node --experimental-strip-types e2e/stubs/notion-stub.ts", + url: NOTION_STUB_ORIGIN, + reuseExistingServer: false, + timeout: 30_000, + }, { // `npm run build` runs `db:migrate` first, which is a no-op with the // blanked DATABASE_URL below, then `next build`. @@ -97,6 +123,9 @@ export default defineConfig({ // is itself asserted in projects.spec.ts, and no test may put bytes in // somebody's real store. BLOB_READ_WRITE_TOKEN: "", + // `next start` is a production build, so the local `.blob-data/` store + // is off anyway; this says so explicitly (src/lib/blob-mode.ts). + BLOB_LOCAL_DISABLE: "1", // Same reasoning for the nightly job: no E2E test should be able to // trigger a real backup. CRON_SECRET: "", @@ -118,6 +147,10 @@ export default defineConfig({ ANTHROPIC_API_KEY: "e2e-stub-key", ANTHROPIC_BASE_URL: `${ANTHROPIC_STUB_ORIGIN}/v1`, AI_GATEWAY_API_KEY: "", + // The mirror's Notion calls go to the stub on this machine (test-only + // override; production never sets it). The token the spec connects with + // is accepted by nothing else, and is encrypted under AUTH_SECRET above. + NOTION_API_BASE_URL: `${NOTION_STUB_ORIGIN}/v1`, // The Workflow SDK's local world (spec §3.7): its queue calls this // server's own /.well-known/workflow routes, so it needs the address // `next start -p 3100` serves on. Runs are kept in their own folder, and diff --git a/v5/scripts/check-spec-coverage.ts b/v5/scripts/check-spec-coverage.ts index c55f6ac..57982a3 100644 --- a/v5/scripts/check-spec-coverage.ts +++ b/v5/scripts/check-spec-coverage.ts @@ -122,6 +122,8 @@ const ACCEPTED: Record = { "env-var:VERCEL_URL": "platform built-in", "env-var:VERCEL_ENV": "platform built-in", "env-var:VERCEL_PROJECT_PRODUCTION_URL": "platform built-in", + "env-var:NOTION_API_BASE_URL": + "test-only: points the mirror's Notion client at the E2E stub (e2e/stubs/notion-stub.ts); production never sets it (v5/AGENTS.md)", "npm-script:spec:coverage": "this script", "npm-script:migrate:resources": "one-off migration tool, not app surface", "npm-script:drop:deprecated-columns": "one-off migration tool, not app surface", diff --git a/v5/src/app/admin/inventory/resource-actions.ts b/v5/src/app/admin/inventory/resource-actions.ts index 0f3655a..9669fd1 100644 --- a/v5/src/app/admin/inventory/resource-actions.ts +++ b/v5/src/app/admin/inventory/resource-actions.ts @@ -8,6 +8,7 @@ import { type ResourceCreatePayload, type ResourceWritePayload, } from "../../../lib/inventory/resource-edits"; +import { requestManualArchive } from "../../../lib/manuals/trigger"; import type { InventoryActionResult } from "./action-result"; import { withToolEdit, type ToolWriteInput } from "./tool-write-context"; @@ -43,18 +44,28 @@ export async function addResource( fileAttachmentIds?: readonly string[]; } ): Promise> { - return withToolEdit(input, (context) => + const result = await withToolEdit(input, (context) => addResourceWrite(context, input.resource, input.fileAttachmentIds ?? []) ); + // A link may be a manual worth keeping a copy of. Only after the write + // landed, and never able to fail it (`requestManualArchive` never throws). + if (result.ok && input.resource.url) await requestManualArchive([result.resourceId]); + return result; } /** Edit a resource — title, type, link, notes, or whether it is published. */ export async function editResource( input: ToolWriteInput & { resourceId: string; patch: ResourcePatch } ): Promise> { - return withToolEdit(input, (context) => + const result = await withToolEdit(input, (context) => editResourceWrite(context, input.resourceId, input.patch) ); + // A new link (or a new type) may mean a manual to copy; the archive skips + // one it already holds, so an unchanged link costs a query and no download. + if (result.ok && (input.patch.url || input.patch.type !== undefined)) { + await requestManualArchive([result.resourceId]); + } + return result; } /** diff --git a/v5/src/app/admin/inventory/tool-write-context.mirror.test.ts b/v5/src/app/admin/inventory/tool-write-context.mirror.test.ts new file mode 100644 index 0000000..28c571a --- /dev/null +++ b/v5/src/app/admin/inventory/tool-write-context.mirror.test.ts @@ -0,0 +1,115 @@ +// @vitest-environment node +import { nextCacheMock } from "../../../../test/mocks/next-cache"; +import { nextHeadersMock, setMockHeaders } from "../../../../test/mocks/next-headers"; + +vi.mock("next/cache", () => nextCacheMock()); +vi.mock("next/headers", () => nextHeadersMock()); + +// The trigger has its own tests; here it is only asked whether it was called. +const mirror = vi.hoisted(() => ({ requestMirrorPush: vi.fn() })); + +vi.mock("../../../lib/mirror/trigger", () => ({ requestMirrorPush: mirror.requestMirrorPush })); + +import { resetAuthForTests } from "../../../lib/auth/config"; +import { readToolRevision } from "../../../lib/data/tools"; +import { getDb, resetDbForTests } from "../../../lib/db/client"; +import { auditEvents, session, tools, units, user } from "../../../lib/db/schema/index"; +import { signInAsNew } from "../../../../test/utils/session"; +import { publish, saveTool } from "./actions"; +import { withToolWrite } from "./tool-write-context"; + +/** + * Every tool-editor write is a mirror trigger (spec §3.8 trigger 1: + * "publishing, and saving an edit call `requestMirrorPush()`") — and only a + * write that landed. `withToolWrite` is the one place that knows both, so it + * is tested directly with a stand-in write, and once more through two real + * actions so the wiring cannot come loose. + */ + +const AUTH_SECRET = "tool-write-mirror-test-secret"; + +let toolId: string; + +beforeEach(async () => { + vi.stubEnv("DATABASE_URL", ""); + vi.stubEnv("AUTH_SECRET", AUTH_SECRET); + vi.stubEnv("AUTH_SUPER_ADMIN_EMAILS", ""); + resetAuthForTests(); + mirror.requestMirrorPush.mockReset().mockResolvedValue(undefined); + + const db = await getDb(); + await db.delete(auditEvents); + await db.delete(units); + await db.delete(tools); + await db.delete(session); + await db.delete(user); + const [row] = await db + .insert(tools) + .values({ slug: "form-4", name: "Form 4", description: "before", published: false }) + .returning({ id: tools.id }); + toolId = row.id; +}); + +afterEach(() => { + resetAuthForTests(); + resetDbForTests(); +}); + +async function asSuperMaker() { + const signedIn = await signInAsNew({ email: "maker@cornell.edu", role: "admin", name: "Luis" }); + setMockHeaders({ cookie: signedIn.cookie }); +} + +async function input() { + return { toolId, expectedRevision: (await readToolRevision(toolId))! }; +} + +describe("withToolWrite and the mirror", () => { + it("requests a push after a write that landed", async () => { + await asSuperMaker(); + + const result = await withToolWrite("tools.edit", await input(), async (context) => ({ + ok: true as const, + revision: context.expectedRevision, + })); + + expect(result.ok).toBe(true); + expect(mirror.requestMirrorPush).toHaveBeenCalledTimes(1); + }); + + it("requests nothing after a refused write", async () => { + await asSuperMaker(); + + const result = await withToolWrite("tools.edit", await input(), async () => ({ + ok: false as const, + error: "conflict" as const, + })); + + expect(result).toEqual({ ok: false, error: "conflict" }); + expect(mirror.requestMirrorPush).not.toHaveBeenCalled(); + }); + + it("requests nothing when the gate refuses, and never runs the write", async () => { + setMockHeaders(); + const write = vi.fn(); + + expect(await withToolWrite("tools.edit", await input(), write)).toEqual({ ok: false, error: "not_signed_in" }); + expect(write).not.toHaveBeenCalled(); + expect(mirror.requestMirrorPush).not.toHaveBeenCalled(); + }); + + it("requests a push from a real save and a real publish, and not from a stale save", async () => { + await asSuperMaker(); + const stale = await input(); + + expect((await saveTool({ ...stale, patch: { description: "after" } })).ok).toBe(true); + expect(mirror.requestMirrorPush).toHaveBeenCalledTimes(1); + + expect((await publish(await input())).ok).toBe(true); + expect(mirror.requestMirrorPush).toHaveBeenCalledTimes(2); + + // The first token is spent: a conflict, nothing written, nothing to mirror. + expect(await saveTool({ ...stale, patch: { description: "again" } })).toMatchObject({ ok: false, error: "conflict" }); + expect(mirror.requestMirrorPush).toHaveBeenCalledTimes(2); + }); +}); diff --git a/v5/src/app/admin/inventory/tool-write-context.ts b/v5/src/app/admin/inventory/tool-write-context.ts index e9c548e..f8b9ab8 100644 --- a/v5/src/app/admin/inventory/tool-write-context.ts +++ b/v5/src/app/admin/inventory/tool-write-context.ts @@ -3,15 +3,20 @@ import { authorizeAdminAction } from "../../../lib/admin/action-gate"; import type { Permission } from "../../../lib/auth/permissions"; import type { Revision } from "../../../lib/data/revision"; import type { InventoryWriteResult } from "../../../lib/inventory/result"; +import { requestMirrorPush } from "../../../lib/mirror/trigger"; import { INVENTORY_PATH, type InventoryActionResult } from "./action-result"; /** * The preamble every tool-editor write shares (spec §5.3, §8). * * Whatever a panel control changes — a field, a unit, a resource, a photo, the - * tool's state — the action behind it is the same three moves: check its own + * tool's state — the action behind it is the same moves: check its own * permission, build the write context out of the caller's identity and the - * token the panel holds, and refresh the review table only if the write landed. + * token the panel holds, and — only if the write landed — refresh the review + * table and ask the Notion mirror to catch up (`requestMirrorPush()`, §3.8 + * trigger 1: "publishing, and saving an edit"). Doing it here is what makes + * every editor write, publish, archive and Looks good a trigger, without each + * action having to remember. * This module owns those moves so `actions.ts` and the three child-section * modules beside it hold nothing but the writes they are named after. * @@ -54,10 +59,15 @@ export async function withToolWrite( }); // Only on a success: a refused write changed nothing, and re-rendering the - // review table for it buys a page of queries for no reason. The catalogue's - // own invalidation happened inside `src/lib/inventory/`, which is the layer - // that knows whether the transaction committed. - if (result.ok) revalidatePath(INVENTORY_PATH); + // review table for it buys a page of queries for no reason — and a push for + // it would mirror nothing. The catalogue's own invalidation happened inside + // `src/lib/inventory/`, which is the layer that knows whether the + // transaction committed. `requestMirrorPush` never throws, so a mirror that + // cannot be told never turns a landed write into a failure (Article 4). + if (result.ok) { + revalidatePath(INVENTORY_PATH); + await requestMirrorPush(); + } return result; } diff --git a/v5/src/app/admin/maintenance/actions.mirror.test.ts b/v5/src/app/admin/maintenance/actions.mirror.test.ts new file mode 100644 index 0000000..825fa7a --- /dev/null +++ b/v5/src/app/admin/maintenance/actions.mirror.test.ts @@ -0,0 +1,87 @@ +// @vitest-environment node +import { nextCacheMock } from "../../../../test/mocks/next-cache"; +import { nextHeadersMock, setMockHeaders } from "../../../../test/mocks/next-headers"; + +vi.mock("next/cache", () => nextCacheMock()); +vi.mock("next/headers", () => nextHeadersMock()); + +// The trigger has its own tests; here it is only asked whether it was called. +const mirror = vi.hoisted(() => ({ requestMirrorPush: vi.fn() })); + +vi.mock("../../../lib/mirror/trigger", () => ({ requestMirrorPush: mirror.requestMirrorPush })); + +import { resetAuthForTests } from "../../../lib/auth/config"; +import { getDb, resetDbForTests } from "../../../lib/db/client"; +import { maintenanceLogs, session, tools, units, user } from "../../../lib/db/schema/index"; +import { signInAsNew } from "../../../../test/utils/session"; +import { updateTicket } from "./actions"; + +/** + * Working a ticket is a mirror trigger (spec §3.8): the mirror carries every + * maintenance log, so a status, priority, assignee or resolution is a change it + * should hold. A refused write changes nothing and asks for nothing, and a + * trigger that fails never turns a landed write into a failure. + */ + +const AUTH_SECRET = "admin-maintenance-mirror-test-secret"; + +let ticketId: string; + +beforeEach(async () => { + vi.stubEnv("DATABASE_URL", ""); + vi.stubEnv("AUTH_SECRET", AUTH_SECRET); + vi.stubEnv("AUTH_SUPER_ADMIN_EMAILS", ""); + vi.stubEnv("LAB_TIMEZONE", "America/New_York"); + resetAuthForTests(); + mirror.requestMirrorPush.mockReset().mockResolvedValue(undefined); + + const db = await getDb(); + await db.delete(maintenanceLogs); + await db.delete(units); + await db.delete(tools); + await db.delete(session); + await db.delete(user); + const [tool] = await db + .insert(tools) + .values({ slug: "trotec", name: "Trotec Speedy 400" }) + .returning({ id: tools.id }); + const [ticket] = await db + .insert(maintenanceLogs) + .values({ title: "Laser bed out of focus", status: "open", priority: "high", toolId: tool.id }) + .returning({ id: maintenanceLogs.id }); + ticketId = ticket.id; +}); + +afterEach(() => { + resetAuthForTests(); + resetDbForTests(); +}); + +it("requests a push after a ticket changes", async () => { + const admin = await signInAsNew({ email: "niti@cornell.edu", role: "admin" }); + setMockHeaders({ cookie: admin.cookie }); + + expect(await updateTicket({ logId: ticketId, patch: { status: "resolved" } })).toEqual({ ok: true }); + expect(mirror.requestMirrorPush).toHaveBeenCalledTimes(1); +}); + +it("requests nothing when the caller is refused", async () => { + const student = await signInAsNew({ email: "casey@cornell.edu", role: "user" }); + setMockHeaders({ cookie: student.cookie }); + + expect(await updateTicket({ logId: ticketId, patch: { status: "closed" } })).toEqual({ + ok: false, + error: "not_permitted", + }); + expect(mirror.requestMirrorPush).not.toHaveBeenCalled(); +}); + +it("requests nothing when the ticket does not exist", async () => { + const admin = await signInAsNew({ email: "niti@cornell.edu", role: "admin" }); + setMockHeaders({ cookie: admin.cookie }); + + const result = await updateTicket({ logId: crypto.randomUUID(), patch: { status: "resolved" } }); + + expect(result.ok).toBe(false); + expect(mirror.requestMirrorPush).not.toHaveBeenCalled(); +}); diff --git a/v5/src/app/admin/maintenance/actions.ts b/v5/src/app/admin/maintenance/actions.ts index 172eaae..b9673c7 100644 --- a/v5/src/app/admin/maintenance/actions.ts +++ b/v5/src/app/admin/maintenance/actions.ts @@ -2,6 +2,7 @@ import { runQueueWrite } from "../../../lib/admin/queue-write"; import { updateMaintenanceLog } from "../../../lib/data/maintenance"; +import { requestMirrorPush } from "../../../lib/mirror/trigger"; import { MAINTENANCE_PATH, type MaintenanceActionResult, @@ -26,6 +27,12 @@ import { * shows unit *status*, which is the tool editor's field and a different write. * Busting the catalogue here would cost a full re-read every time somebody * ticked a box. + * + * **But it does tell the Notion mirror.** The mirror carries every maintenance + * log (§3.8), so a ticket's status, priority, assignee or resolution is a + * change it should hold. `requestMirrorPush()` runs after the write commits, + * never throws, and coalesces: a reviewer clearing ten tickets starts one + * push two minutes later, not ten. A refused write never reaches it. */ /** Names this surface in the console line a failure leaves behind. */ @@ -49,5 +56,9 @@ export async function updateTicket(input: { surface: SURFACE, write: (identity) => updateMaintenanceLog(input.logId, input.patch, { actorUserId: identity.userId }), + afterCommit: async () => { + await requestMirrorPush(); + return undefined; + }, }); } diff --git a/v5/src/app/admin/mirror/action-result.ts b/v5/src/app/admin/mirror/action-result.ts new file mode 100644 index 0000000..59d1faa --- /dev/null +++ b/v5/src/app/admin/mirror/action-result.ts @@ -0,0 +1,96 @@ +import type { AdminActionWarning, AdminGateError } from "../../../lib/admin/action-result"; +import type { MirrorEntity } from "../../../lib/db/schema/vocabulary"; +import { MIRROR_SETUP_ERRORS, type MappingProblem, type MirrorSetupError } from "../../../lib/mirror/types"; + +/** + * What `/admin/mirror`'s server actions answer, and where they live. + * + * Directive-free for the reason every admin surface's result module is: a + * `"use server"` module may export only async functions, and the mirror's + * islands render these codes without importing the endpoints to get at their + * shape. Its only runtime import is the client-safe `lib/mirror/types.ts`. + */ + +/** The page every action refreshes. */ +export const MIRROR_PATH = "/admin/mirror"; + +/** + * Why a mirror action did nothing, or not all of it. + * + * Two families, rendered from two message namespaces: the gate's codes (and + * `invalid_field`, a body that did not parse) from `admin.errors.` like + * every other admin surface, and the mirror's own from + * `admin.mirror.errors.`. The names do not overlap, so one union is + * enough; {@link mirrorErrorMessageKey} says which namespace a code is in. + */ +export type MirrorActionError = AdminGateError | "invalid_field" | MirrorSetupError; + +/** A refusal, with whatever the island needs to explain it. */ +export interface MirrorActionFailure { + ok: false; + error: MirrorActionError; + /** `sync_too_soon`: seconds until Sync now is allowed again. */ + retryAfterSeconds?: number; + /** `saveMapping`: which pasted ids did not validate, and why. */ + problems?: MappingProblem[]; + /** + * `createDatabases`: the databases made before it stopped. They exist in + * Notion and are in the mapping — a failure that still changed something, + * which the page must say rather than hide (Article 4). + */ + created?: MirrorEntity[]; + /** `createDatabases`: the entity it stopped on, when there was one. */ + entity?: MirrorEntity | null; +} + +/** A change that landed. A lost audit event rides here as `warning` (§4.11). */ +export type MirrorActionResult = { ok: true; warning?: AdminActionWarning } | MirrorActionFailure; + +/** **Test connection**: the page's title, and nothing stored. */ +export type MirrorTestResult = { ok: true; pageId: string; title: string | null } | MirrorActionFailure; + +/** **Connect**: the page it connected to. */ +export type MirrorConnectActionResult = + | { ok: true; title: string | null; warning?: AdminActionWarning } + | MirrorActionFailure; + +/** **Create databases**: which were made and which already existed. */ +export type MirrorCreateResult = + | { ok: true; created: MirrorEntity[]; kept: MirrorEntity[] } + | MirrorActionFailure; + +/** What Connect and Test connection send. The token travels in, never back out. */ +export interface MirrorConnectInput { + token: string; + pageUrl: string; +} + +/** Pasted database ids, by entity. An entity left out is not changed. */ +export type MirrorMappingInput = Partial>; + +/** + * The bundle the page hands its islands. Built by the page rather than + * exported from `actions.ts`, because a `"use server"` module may export only + * async functions. None of them takes a mirror id: the server finds the + * mirror from the session, so nobody can address somebody else's (§8). + */ +export interface MirrorActions { + testConnection: (input: MirrorConnectInput) => Promise; + connect: (input: MirrorConnectInput) => Promise; + createDatabases: () => Promise; + saveMapping: (input: MirrorMappingInput) => Promise; + syncNow: () => Promise; + setPaused: (input: { paused: boolean }) => Promise; + disconnect: () => Promise; +} + +const SETUP_ERRORS: ReadonlySet = new Set(MIRROR_SETUP_ERRORS); + +/** + * The message key, under `admin`, that explains `code`: `mirror.errors.` + * for the mirror's own codes, `errors.` for the ones every admin surface + * shares. + */ +export function mirrorErrorMessageKey(code: MirrorActionError): string { + return SETUP_ERRORS.has(code) ? `mirror.errors.${code}` : `errors.${code}`; +} diff --git a/v5/src/app/admin/mirror/actions.test.ts b/v5/src/app/admin/mirror/actions.test.ts new file mode 100644 index 0000000..e11ca02 --- /dev/null +++ b/v5/src/app/admin/mirror/actions.test.ts @@ -0,0 +1,546 @@ +// @vitest-environment node +import { nextCacheMock } from "../../../../test/mocks/next-cache"; +import { nextHeadersMock, setMockHeaders } from "../../../../test/mocks/next-headers"; + +vi.mock("next/cache", () => nextCacheMock()); +vi.mock("next/headers", () => nextHeadersMock()); + +/** Withhold or grant one permission — see `admin/corrections/actions.test.ts`. */ +const override = vi.hoisted(() => ({ permissions: null as Set | null })); + +vi.mock("../../../lib/auth/permissions", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + can: (subject: Parameters[0], permission: string) => + override.permissions + ? override.permissions.has(permission) + : actual.can(subject, permission as Parameters[1]), + }; +}); + +/** The audit insert is a second statement, after the change; it can fail on its own. */ +const audit = vi.hoisted(() => ({ failing: false })); + +vi.mock("../../../lib/data/audit", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + recordAuditEvent: async (event: Parameters[0]) => { + if (audit.failing) throw new Error("connection terminated unexpectedly"); + return actual.recordAuditEvent(event); + }, + }; +}); + +/** A connect that throws, to prove a thrown error's text is scrubbed before it is logged. */ +const connectThrow = vi.hoisted(() => ({ message: null as string | null })); + +vi.mock("../../../lib/mirror/connect", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + connectMirror: async (...args: Parameters) => { + if (connectThrow.message) throw new Error(connectThrow.message); + return actual.connectMirror(...args); + }, + }; +}); + +// Starting a workflow is Part B's; creating and validating databases is Part +// A's. Both are stubbed here so these tests are about the endpoints: who may +// call them, which mirror they reach, and what they answer. +vi.mock("../../../lib/mirror/start", () => ({ + syncMirrorNow: vi.fn(), + startMirrorPush: vi.fn(), + startCoalescedPush: vi.fn(), +})); + +vi.mock("../../../lib/mirror/databases", () => ({ + ensureMirrorDatabases: vi.fn(), + applyPastedMapping: vi.fn(), +})); + +import { eq } from "drizzle-orm"; +import { revalidatePath } from "next/cache"; +import { server } from "../../../../test/msw/server"; +// Aliased: the name starts with `use`, which eslint's rules-of-hooks reads as a React hook. +import { useNotionFake as installNotionFake } from "../../../../test/msw/notion-mirror"; +import { createNotionFake, type NotionFake } from "../../../../test/fakes/notion-fake"; +import { signInAsNew, type SignedInSession } from "../../../../test/utils/session"; +import { resetAuthForTests } from "../../../lib/auth/config"; +import { saveMirrorConnection, setMirrorMapping } from "../../../lib/data/mirrors"; +import { getDb, resetDbForTests } from "../../../lib/db/client"; +import { auditEvents, notionMirrors, session, user } from "../../../lib/db/schema/index"; +import type { Db } from "../../../lib/db/types"; +import { applyPastedMapping, ensureMirrorDatabases } from "../../../lib/mirror/databases"; +import { syncMirrorNow } from "../../../lib/mirror/start"; +import { decryptMirrorToken, encryptMirrorToken } from "../../../lib/mirror/token-crypto"; +import { + connect, + createDatabases, + disconnect, + saveMapping, + setPaused, + syncNow, + testConnection, +} from "./actions"; + +/** + * The mirror page's endpoints (spec §3.8, §5.8, §8), called directly with no + * page — a server action is a POST endpoint with a generated name. + * + * What is under test: each action refuses anybody without `mirror.manage`; a + * mirror is reachable only by its owner, because no action takes its id; a + * token is validated by one read before anything is stored; the token appears + * in no result and no console line; and a change that landed minus its audit + * event is a warning on a success, never a failure. + */ + +const AUTH_SECRET = "admin-mirror-actions-test-secret"; +const TOKEN = "ntn_ACTIONStestToken0123456789abcdef"; +const PAGE_ID = "0f5e4a3c-1111-2222-3333-44445555aaaa"; +const PAGE_URL = `https://www.notion.so/acme/MakerLab-Tools-mirror-${PAGE_ID.replace(/-/g, "")}`; +const TITLE = "MakerLab Tools — mirror"; +const TOOLS_DB = "1a2b3c4d-0000-4000-8000-000000000001"; + +let db: Db; +let fake: NotionFake; +let consoleLines: string[]; + +beforeEach(async () => { + vi.stubEnv("DATABASE_URL", ""); + vi.stubEnv("AUTH_SECRET", AUTH_SECRET); + vi.stubEnv("AUTH_SUPER_ADMIN_EMAILS", ""); + resetAuthForTests(); + override.permissions = null; + audit.failing = false; + connectThrow.message = null; + vi.mocked(revalidatePath).mockClear(); + vi.mocked(syncMirrorNow).mockReset(); + vi.mocked(ensureMirrorDatabases).mockReset(); + vi.mocked(applyPastedMapping).mockReset(); + + fake = createNotionFake({ token: TOKEN, pages: [{ id: PAGE_ID, title: TITLE }] }); + installNotionFake(server, fake); + + consoleLines = []; + for (const method of ["log", "info", "warn", "error", "debug"] as const) { + vi.spyOn(console, method).mockImplementation((...args: unknown[]) => { + consoleLines.push(args.map((arg) => (arg instanceof Error ? `${arg.message}\n${arg.stack}` : String(arg))).join(" ")); + }); + } + + db = await getDb(); + await db.delete(notionMirrors); + await db.delete(auditEvents); + await db.delete(session); + await db.delete(user); +}); + +afterEach(() => { + // Whatever a test did, the token never reached the console (§8, §10). + expect(consoleLines.join("\n")).not.toContain(TOKEN); + vi.restoreAllMocks(); + override.permissions = null; + audit.failing = false; + resetAuthForTests(); +}); + +afterAll(() => { + resetDbForTests(); +}); + +async function asAdmin(email = `admin-${Math.random().toString(36).slice(2)}@cornell.edu`) { + const signedIn = await signInAsNew({ email, role: "admin" }); + setMockHeaders({ cookie: signedIn.cookie }); + return signedIn; +} + +async function mirrorOf(owner: SignedInSession) { + const [row] = await db.select().from(notionMirrors).where(eq(notionMirrors.ownerUserId, owner.user.id)); + return row ?? null; +} + +/** A connected mirror, written straight to the table. */ +async function seedMirror(owner: SignedInSession, mapping: Record = { tools: TOOLS_DB }) { + const { mirror } = await saveMirrorConnection( + { + ownerUserId: owner.user.id, + tokenCiphertext: encryptMirrorToken(TOKEN, AUTH_SECRET), + parentPageId: PAGE_ID, + parentPageTitle: TITLE, + }, + { db } + ); + if (Object.keys(mapping).length) await setMirrorMapping(mirror.id, mapping, { db }); + return mirror; +} + +/** Every action, once, with a well-formed body. */ +function everyAction() { + const body = { token: TOKEN, pageUrl: PAGE_URL }; + return [ + ["testConnection", () => testConnection(body)], + ["connect", () => connect(body)], + ["createDatabases", () => createDatabases()], + ["saveMapping", () => saveMapping({ tools: TOOLS_DB })], + ["syncNow", () => syncNow()], + ["setPaused", () => setPaused({ paused: true })], + ["disconnect", () => disconnect()], + ] as const; +} + +describe("the gate", () => { + it("refuses an anonymous caller on every action, and touches nothing", async () => { + setMockHeaders(); + for (const [name, call] of everyAction()) { + expect(await call(), name).toEqual({ ok: false, error: "not_signed_in" }); + } + expect(await db.select().from(notionMirrors)).toHaveLength(0); + expect(fake.requests).toHaveLength(0); + }); + + it("refuses a student on every action", async () => { + const student = await signInAsNew({ email: "casey@cornell.edu", role: "user" }); + setMockHeaders({ cookie: student.cookie }); + for (const [name, call] of everyAction()) { + expect(await call(), name).toEqual({ ok: false, error: "not_permitted" }); + } + expect(await db.select().from(notionMirrors)).toHaveLength(0); + expect(fake.requests).toHaveLength(0); + }); + + it("checks mirror.manage itself, not some other admin permission", async () => { + const admin = await asAdmin(); + await seedMirror(admin); + + // An account holding every other admin permission is still refused. + override.permissions = new Set([ + "tools.edit", + "tools.publish", + "tools.approve", + "users.manage", + "maintenance.manage", + "feedback.manage", + "projects.moderate", + ]); + for (const [name, call] of everyAction()) { + expect(await call(), name).toEqual({ ok: false, error: "not_permitted" }); + } + + override.permissions = new Set(["mirror.manage"]); + expect(await setPaused({ paused: true })).toEqual({ ok: true }); + }); + + it("rate-limits the setup calls that spend the admin's Notion token", async () => { + await asAdmin(); + for (let i = 0; i < 10; i += 1) { + expect(await testConnection({ token: TOKEN, pageUrl: PAGE_URL })).toMatchObject({ ok: true }); + } + expect(await testConnection({ token: TOKEN, pageUrl: PAGE_URL })).toEqual({ ok: false, error: "rate_limited" }); + expect(await connect({ token: TOKEN, pageUrl: PAGE_URL })).toEqual({ ok: false, error: "rate_limited" }); + expect(fake.requests).toHaveLength(10); + }); + + it("refuses a body it does not recognise, including one that names a mirror", async () => { + await asAdmin(); + expect(await setPaused({ paused: "yes" })).toEqual({ ok: false, error: "invalid_field" }); + expect(await setPaused({ paused: true, mirrorId: crypto.randomUUID() })).toEqual({ + ok: false, + error: "invalid_field", + }); + expect(await syncNow({ mirrorId: crypto.randomUUID() })).toEqual({ ok: false, error: "invalid_field" }); + expect(await testConnection("not an object")).toEqual({ ok: false, error: "invalid_field" }); + }); +}); + +describe("Test connection and Connect", () => { + it("tests the connection, answers the page's title, and stores nothing", async () => { + const admin = await asAdmin(); + + expect(await testConnection({ token: TOKEN, pageUrl: PAGE_URL })).toEqual({ + ok: true, + pageId: PAGE_ID, + title: TITLE, + }); + expect(await mirrorOf(admin)).toBeNull(); + expect(await db.select().from(auditEvents)).toHaveLength(0); + expect(vi.mocked(revalidatePath)).not.toHaveBeenCalled(); + }); + + it("connects: the token encrypted, the event recorded with the page id only, the page refreshed", async () => { + const admin = await asAdmin(); + + expect(await connect({ token: TOKEN, pageUrl: PAGE_URL })).toEqual({ ok: true, title: TITLE }); + + const row = await mirrorOf(admin); + expect(row?.parentPageId).toBe(PAGE_ID); + expect(decryptMirrorToken(row?.tokenCiphertext as Uint8Array, AUTH_SECRET)).toBe(TOKEN); + + const [event] = await db.select().from(auditEvents); + expect(event).toMatchObject({ + action: "mirror.connected", + actorUserId: admin.user.id, + subjectType: "mirror", + subjectId: row?.id, + detail: { parentPageId: PAGE_ID }, + }); + expect(JSON.stringify(event)).not.toContain(TOKEN); + expect(vi.mocked(revalidatePath)).toHaveBeenCalledWith("/admin/mirror"); + }); + + it("stores nothing and records nothing when the read fails", async () => { + const admin = await asAdmin(); + + expect(await connect({ token: `${TOKEN}WRONG`, pageUrl: PAGE_URL })).toEqual({ + ok: false, + error: "unauthorized", + }); + expect(await connect({ token: TOKEN, pageUrl: crypto.randomUUID() })).toEqual({ + ok: false, + error: "page_not_found", + }); + expect(await connect({ token: "short", pageUrl: PAGE_URL })).toEqual({ ok: false, error: "invalid_token" }); + expect(await connect({ token: TOKEN, pageUrl: "https://example.com/" })).toEqual({ + ok: false, + error: "invalid_page", + }); + + expect(await mirrorOf(admin)).toBeNull(); + expect(await db.select().from(auditEvents)).toHaveLength(0); + expect(vi.mocked(revalidatePath)).not.toHaveBeenCalled(); + }); + + it("keeps the connection and warns when the audit event cannot be written", async () => { + const admin = await asAdmin(); + audit.failing = true; + + expect(await connect({ token: TOKEN, pageUrl: PAGE_URL })).toEqual({ + ok: true, + title: TITLE, + warning: "audit_unavailable", + }); + expect(await mirrorOf(admin)).not.toBeNull(); + }); + + it("puts the token in no result, and scrubs it from the line a thrown error leaves", async () => { + await asAdmin(); + + const results = [ + await testConnection({ token: TOKEN, pageUrl: PAGE_URL }), + await testConnection({ token: `${TOKEN}WRONG`, pageUrl: PAGE_URL }), + await connect({ token: TOKEN, pageUrl: crypto.randomUUID() }), + await connect({ token: TOKEN, pageUrl: PAGE_URL }), + ]; + for (const result of results) expect(JSON.stringify(result)).not.toContain(TOKEN); + + connectThrow.message = `boom while holding ${TOKEN} for casey@cornell.edu`; + expect(await connect({ token: TOKEN, pageUrl: PAGE_URL })).toEqual({ ok: false, error: "failed" }); + const logged = consoleLines.join("\n"); + expect(logged).toContain("[admin/mirror] the action failed"); + expect(logged).not.toContain("casey@cornell.edu"); + expect(logged).not.toContain(AUTH_SECRET); + // `afterEach` asserts the token itself is absent. + }); + + it("connects nothing when AUTH_SECRET is unset", async () => { + const admin = await asAdmin(); + vi.stubEnv("AUTH_SECRET", ""); + // Sessions are signed with the same secret the token key is derived from, + // so with it gone nobody is signed in and the gate refuses first. + // `connectMirror`'s own `key_unavailable` is covered in lib/mirror/connect.test.ts. + expect(await connect({ token: TOKEN, pageUrl: PAGE_URL })).toEqual({ ok: false, error: "not_signed_in" }); + expect(await mirrorOf(admin)).toBeNull(); + }); +}); + +describe("one admin cannot reach another's mirror", () => { + it("answers not_connected to admin B for everything, and admin A's mirror is untouched", async () => { + const adminA = await asAdmin("isaac@cornell.edu"); + const mirrorA = await seedMirror(adminA); + const before = await mirrorOf(adminA); + + await asAdmin("niti@cornell.edu"); + vi.mocked(syncMirrorNow).mockResolvedValue({ ok: false, code: "not_connected" }); + + expect(await createDatabases()).toEqual({ ok: false, error: "not_connected" }); + expect(await saveMapping({ tools: TOOLS_DB })).toEqual({ ok: false, error: "not_connected" }); + expect(await syncNow()).toEqual({ ok: false, error: "not_connected" }); + expect(await setPaused({ paused: true })).toEqual({ ok: false, error: "not_connected" }); + expect(await disconnect()).toEqual({ ok: false, error: "not_connected" }); + + expect(vi.mocked(ensureMirrorDatabases)).not.toHaveBeenCalled(); + expect(vi.mocked(applyPastedMapping)).not.toHaveBeenCalled(); + expect(vi.mocked(syncMirrorNow)).not.toHaveBeenCalled(); + + const after = await mirrorOf(adminA); + expect(after?.pausedAt).toBeNull(); + expect(after?.tokenCiphertext).toEqual(before?.tokenCiphertext); + expect(after?.mapping).toEqual({ tools: TOOLS_DB }); + expect(after?.id).toBe(mirrorA.id); + }); + + it("gives admin B a mirror of B's own when B connects", async () => { + const adminA = await asAdmin("isaac@cornell.edu"); + const mirrorA = await seedMirror(adminA); + + const adminB = await asAdmin("niti@cornell.edu"); + expect(await connect({ token: TOKEN, pageUrl: PAGE_URL })).toMatchObject({ ok: true }); + + const mirrorB = await mirrorOf(adminB); + expect(mirrorB?.id).not.toBe(mirrorA.id); + expect(mirrorB?.mapping).toEqual({}); + expect((await mirrorOf(adminA))?.mapping).toEqual({ tools: TOOLS_DB }); + }); +}); + +describe("Create databases and the pasted mapping", () => { + it("creates databases for the caller's own mirror and says which", async () => { + const admin = await asAdmin(); + const mirror = await seedMirror(admin, {}); + vi.mocked(ensureMirrorDatabases).mockResolvedValue({ + ok: true, + created: ["categories", "tools"], + kept: [], + mapping: {}, + }); + + expect(await createDatabases()).toEqual({ ok: true, created: ["categories", "tools"], kept: [] }); + expect(vi.mocked(ensureMirrorDatabases)).toHaveBeenCalledWith(mirror.id); + expect(vi.mocked(revalidatePath)).toHaveBeenCalledWith("/admin/mirror"); + }); + + it("reports a create that stopped part-way, and refreshes the page because it still made some", async () => { + const admin = await asAdmin(); + await seedMirror(admin, {}); + vi.mocked(ensureMirrorDatabases).mockResolvedValue({ + ok: false, + code: "notion_unavailable", + created: ["categories"], + entity: "locations", + }); + + expect(await createDatabases()).toEqual({ + ok: false, + error: "notion_unavailable", + created: ["categories"], + entity: "locations", + }); + expect(vi.mocked(revalidatePath)).toHaveBeenCalledWith("/admin/mirror"); + }); + + it("refuses without a token, and never asks Notion", async () => { + const admin = await asAdmin(); + await seedMirror(admin); + await disconnect(); + + expect(await createDatabases()).toEqual({ ok: false, error: "not_connected" }); + expect(await saveMapping({ tools: TOOLS_DB })).toEqual({ ok: false, error: "not_connected" }); + expect(vi.mocked(ensureMirrorDatabases)).not.toHaveBeenCalled(); + expect(vi.mocked(applyPastedMapping)).not.toHaveBeenCalled(); + }); + + it("turns pasted URLs into ids before checking them, and sends only what was pasted", async () => { + const admin = await asAdmin(); + const mirror = await seedMirror(admin, {}); + vi.mocked(applyPastedMapping).mockResolvedValue({ ok: true, mapping: { tools: TOOLS_DB } }); + + expect( + await saveMapping({ tools: `https://www.notion.so/acme/${TOOLS_DB.replace(/-/g, "")}?v=abc`, units: " " }) + ).toEqual({ ok: true }); + expect(vi.mocked(applyPastedMapping)).toHaveBeenCalledWith(mirror.id, { tools: TOOLS_DB }); + }); + + it("refuses a paste that is not a database id without asking Notion", async () => { + const admin = await asAdmin(); + await seedMirror(admin, {}); + + expect(await saveMapping({ tools: "not an id", units: TOOLS_DB })).toEqual({ + ok: false, + error: "invalid_database_id", + problems: [{ entity: "tools", code: "invalid_database_id" }], + }); + expect(await saveMapping({})).toEqual({ ok: false, error: "invalid_database_id", problems: [] }); + expect(await saveMapping({ widgets: TOOLS_DB })).toEqual({ ok: false, error: "invalid_field" }); + expect(vi.mocked(applyPastedMapping)).not.toHaveBeenCalled(); + }); + + it("passes the per-entity problems through, and refreshes nothing", async () => { + const admin = await asAdmin(); + await seedMirror(admin, {}); + const problems = [{ entity: "tools" as const, code: "schema_mismatch" as const, missing: ["Published"] }]; + vi.mocked(applyPastedMapping).mockResolvedValue({ ok: false, code: "schema_mismatch", problems }); + + expect(await saveMapping({ tools: TOOLS_DB })).toEqual({ ok: false, error: "schema_mismatch", problems }); + expect(vi.mocked(revalidatePath)).not.toHaveBeenCalled(); + }); +}); + +describe("Sync now, Pause and Disconnect", () => { + it("syncs the caller's own mirror", async () => { + const admin = await asAdmin(); + await seedMirror(admin); + vi.mocked(syncMirrorNow).mockResolvedValue({ ok: true }); + + expect(await syncNow()).toEqual({ ok: true }); + expect(vi.mocked(syncMirrorNow)).toHaveBeenCalledWith(admin.user.id); + expect(vi.mocked(revalidatePath)).toHaveBeenCalledWith("/admin/mirror"); + }); + + it("passes the 15-minute refusal through, with the time left", async () => { + const admin = await asAdmin(); + await seedMirror(admin); + vi.mocked(syncMirrorNow).mockResolvedValue({ ok: false, code: "sync_too_soon", retryAfterSeconds: 540 }); + + expect(await syncNow()).toEqual({ ok: false, error: "sync_too_soon", retryAfterSeconds: 540 }); + expect(vi.mocked(revalidatePath)).not.toHaveBeenCalled(); + }); + + it("passes a failed start through as start_failed", async () => { + const admin = await asAdmin(); + await seedMirror(admin); + vi.mocked(syncMirrorNow).mockResolvedValue({ ok: false, code: "start_failed" }); + expect(await syncNow()).toEqual({ ok: false, error: "start_failed" }); + }); + + it("pauses and resumes the caller's mirror", async () => { + const admin = await asAdmin(); + await seedMirror(admin); + + expect(await setPaused({ paused: true })).toEqual({ ok: true }); + expect((await mirrorOf(admin))?.pausedAt).toBeInstanceOf(Date); + expect(await setPaused({ paused: false })).toEqual({ ok: true }); + expect((await mirrorOf(admin))?.pausedAt).toBeNull(); + }); + + it("disconnects: forgets the token, keeps the mapping, records the event", async () => { + const admin = await asAdmin(); + const mirror = await seedMirror(admin); + + expect(await disconnect()).toEqual({ ok: true }); + + const row = await mirrorOf(admin); + expect(row?.tokenCiphertext).toBeNull(); + expect(row?.mapping).toEqual({ tools: TOOLS_DB }); + const [event] = await db.select().from(auditEvents); + expect(event).toMatchObject({ + action: "mirror.disconnected", + actorUserId: admin.user.id, + subjectType: "mirror", + subjectId: mirror.id, + }); + + // Nothing is left to disconnect. + expect(await disconnect()).toEqual({ ok: false, error: "not_connected" }); + }); + + it("keeps the disconnect and warns when the audit event cannot be written", async () => { + const admin = await asAdmin(); + await seedMirror(admin); + audit.failing = true; + + expect(await disconnect()).toEqual({ ok: true, warning: "audit_unavailable" }); + expect((await mirrorOf(admin))?.tokenCiphertext).toBeNull(); + }); +}); diff --git a/v5/src/app/admin/mirror/actions.ts b/v5/src/app/admin/mirror/actions.ts new file mode 100644 index 0000000..8ca9111 --- /dev/null +++ b/v5/src/app/admin/mirror/actions.ts @@ -0,0 +1,283 @@ +"use server"; + +import { revalidatePath } from "next/cache"; +import { z } from "zod"; +import { authorizeAdminAction } from "../../../lib/admin/action-gate"; +import { record, warn } from "../../../lib/admin/audit-warning"; +import type { Identity } from "../../../lib/auth/identity"; +import { disconnectMirror, getMirrorForOwner, setMirrorPaused, type MirrorRecord } from "../../../lib/data/mirrors"; +import { MIRROR_ENTITY, type MirrorEntity } from "../../../lib/db/schema/vocabulary"; +import { connectMirror, mirrorTokenSchema, testMirrorConnection } from "../../../lib/mirror/connect"; +import { applyPastedMapping, ensureMirrorDatabases } from "../../../lib/mirror/databases"; +import { scrubSecrets } from "../../../lib/mirror/notion-client"; +import { parseNotionId } from "../../../lib/mirror/notion-id"; +import { syncMirrorNow } from "../../../lib/mirror/start"; +import { MIRROR_SETUP_TIER, rateLimitAsync } from "../../../lib/rate-limit"; +import { + MIRROR_PATH, + type MirrorActionFailure, + type MirrorActionResult, + type MirrorConnectActionResult, + type MirrorCreateResult, + type MirrorTestResult, +} from "./action-result"; + +/** + * The mirror page's endpoints (spec §3.8, §5.8, §8). + * + * **A mirror is its owner's, and nothing here takes its id.** Every action + * finds the mirror from the session's `userId` (`getMirrorForOwner`), so there + * is no parameter through which one admin could read, run, pause or disconnect + * another's (§8 "A mirror can be read, run, paused and disconnected only by its + * owner"). A body carrying a `mirrorId` is refused as `invalid_field` by the + * strict schemas below rather than ignored. + * + * **Each action checks itself**, in the order every admin action does: + * `authorizeAdminAction("mirror.manage")` (identity, `ADMIN_ACTION_TIER`, the + * permission), then — for the four setup calls that spend the admin's Notion + * token (test, connect, create databases, save mapping) — `MIRROR_SETUP_TIER`, + * then the input. A server action is a POST endpoint with a generated name; the + * page having checked `mirror.manage` is evidence of nothing. + * + * **The token comes in and never goes back out.** No result carries it, no + * refusal is built from it, and a thrown error is logged only after + * `scrubSecrets` has taken the token (and anything token- or email-shaped) out + * of its text (§8, §10). + * + * **Refusals are values** rendered from `admin.errors.` or + * `admin.mirror.errors.` (`mirrorErrorMessageKey`). A change that landed + * minus its audit event is `{ ok: true, warning: "audit_unavailable" }`, never + * a failure (§4.11). Only async exports: the shapes and the path live in + * `./action-result.ts`. + */ + +/** Names this surface in the console line a failed write leaves behind. */ +const SURFACE = "admin/mirror"; + +// ── Input shapes ──────────────────────────────────────────────────── + +const connectInput = z.strictObject({ + token: z.string().max(4096), + pageUrl: z.string().max(4096), +}); + +const pausedInput = z.strictObject({ paused: z.boolean() }); + +const mappingInput = z.strictObject( + Object.fromEntries(MIRROR_ENTITY.map((entity) => [entity, z.string().max(2048).optional()])) as Record< + MirrorEntity, + z.ZodOptional + > +); + +const noInput = z.undefined(); + +// ── Actions ───────────────────────────────────────────────────────── + +/** **Test connection**: read the page with the pasted token and show its title. Stores nothing. */ +export async function testConnection(input: unknown): Promise { + const options = { setup: true, refresh: false, schema: connectInput, input, secret: tokenOf(input) }; + return run(options, async (_identity, parsed) => { + const tested = await testMirrorConnection(parsed.token, parsed.pageUrl); + return tested.ok ? { ok: true, pageId: tested.pageId, title: tested.title } : { ok: false, error: tested.code }; + }); +} + +/** + * **Connect**: the same read, then store the token encrypted (§8: validated by + * one read, never stored unverified). Reconnecting keeps the mapping and the + * pages. Audited as `mirror.connected`, with the page id and nothing else. + */ +export async function connect(input: unknown): Promise { + return run({ setup: true, schema: connectInput, input, secret: tokenOf(input) }, async (identity, parsed) => { + const connected = await connectMirror(identity.userId, parsed.token, parsed.pageUrl); + if (!connected.ok) return { ok: false, error: connected.code }; + + const recorded = await record( + { + actorUserId: identity.userId, + action: "mirror.connected", + subjectType: "mirror", + subjectId: connected.mirror.id, + detail: { parentPageId: connected.pageId }, + }, + SURFACE + ); + return { ok: true, title: connected.title, ...warn(undefined, recorded) }; + }); +} + +/** + * **Create databases**: make every mapped-but-missing database under the + * connected page (§3.8, §5.8). A failure part-way still made some — they are + * in the mapping, and the answer names them. + */ +export async function createDatabases(input?: unknown): Promise { + return run({ setup: true, schema: noInput, input }, async (identity) => { + const mirror = await ownMirror(identity); + if (!mirror?.hasToken) return { ok: false, error: "not_connected" }; + + const ensured = await ensureMirrorDatabases(mirror.id); + if (ensured.ok) return { ok: true, created: ensured.created, kept: ensured.kept }; + return { ok: false, error: ensured.code, created: ensured.created, entity: ensured.entity }; + }); +} + +/** + * **Save** pasted database ids: each is checked against the schema it must + * have, and nothing is saved unless every one passes. The problems come back + * per entity, naming the missing and wrong-typed properties. + */ +export async function saveMapping(input: unknown): Promise { + return run({ setup: true, schema: mappingInput, input }, async (identity, parsed) => { + const pasted: Partial> = {}; + const unparsable: MirrorEntity[] = []; + for (const entity of MIRROR_ENTITY) { + const raw = parsed[entity]?.trim(); + if (!raw) continue; + const id = parseNotionId(raw); + if (id) pasted[entity] = id; + else unparsable.push(entity); + } + if (unparsable.length > 0) { + return { + ok: false, + error: "invalid_database_id", + problems: unparsable.map((entity) => ({ entity, code: "invalid_database_id" as const })), + }; + } + if (Object.keys(pasted).length === 0) return { ok: false, error: "invalid_database_id", problems: [] }; + + const mirror = await ownMirror(identity); + if (!mirror?.hasToken) return { ok: false, error: "not_connected" }; + + const applied = await applyPastedMapping(mirror.id, pasted); + return applied.ok ? { ok: true } : { ok: false, error: applied.code, problems: applied.problems }; + }); +} + +/** + * **Sync now**: one push per mirror per 15 minutes (§8). The refusal carries + * how long is left, which the page shows beside the disabled button. + */ +export async function syncNow(input?: unknown): Promise { + return run({ setup: false, schema: noInput, input }, async (identity) => { + const mirror = await ownMirror(identity); + if (!mirror?.hasToken) return { ok: false, error: "not_connected" }; + + const synced = await syncMirrorNow(identity.userId); + if (synced.ok) return { ok: true }; + return synced.retryAfterSeconds !== undefined + ? { ok: false, error: synced.code, retryAfterSeconds: synced.retryAfterSeconds } + : { ok: false, error: synced.code }; + }); +} + +/** **Pause** / **Resume**. A paused mirror is skipped by every trigger. Not audited (§4.11). */ +export async function setPaused(input: unknown): Promise { + return run({ setup: false, schema: pausedInput, input }, async (identity, parsed) => { + const updated = await setMirrorPaused(identity.userId, parsed.paused); + return updated ? { ok: true } : { ok: false, error: "not_connected" }; + }); +} + +/** + * **Disconnect** forgets the token (§3.8). The mapping and the pages stay, so + * connecting again updates the same Notion pages instead of duplicating them. + * Audited as `mirror.disconnected`. + */ +export async function disconnect(input?: unknown): Promise { + return run({ setup: false, schema: noInput, input }, async (identity) => { + const mirror = await ownMirror(identity); + if (!mirror?.hasToken) return { ok: false, error: "not_connected" }; + if (!(await disconnectMirror(identity.userId))) return { ok: false, error: "not_connected" }; + + const recorded = await record( + { + actorUserId: identity.userId, + action: "mirror.disconnected", + subjectType: "mirror", + subjectId: mirror.id, + }, + SURFACE + ); + return { ok: true, ...warn(undefined, recorded) }; + }); +} + +// ── Internals ─────────────────────────────────────────────────────── + +/** An identity the gate let through, with the user id it is certain to have. */ +type OwnerIdentity = Identity & { userId: string }; + +/** The caller's own mirror — the only one any action here can reach. */ +function ownMirror(identity: OwnerIdentity): Promise { + return getMirrorForOwner(identity.userId); +} + +/** The token in a raw body, if there is one — only so it can be scrubbed from a log line. */ +function tokenOf(input: unknown): string | undefined { + if (!input || typeof input !== "object") return undefined; + const token = (input as { token?: unknown }).token; + if (typeof token !== "string") return undefined; + const parsed = mirrorTokenSchema.safeParse(token); + return parsed.success ? parsed.data : token; +} + +/** A thrown error as one line that holds no token, secret or email. */ +function describe(error: unknown, secret: string | undefined): string { + const text = error instanceof Error ? `${error.name}: ${error.message}` : String(error); + const secrets = [secret, secret?.trim(), process.env.AUTH_SECRET].filter((value): value is string => Boolean(value)); + return scrubSecrets(text, secrets).slice(0, 500); +} + +interface RunOptions

{ + /** Spends the admin's Notion token, so it also passes `MIRROR_SETUP_TIER`. */ + setup: boolean; + schema: z.ZodType

; + input: unknown; + /** Scrubbed from the console line if the action throws. */ + secret?: string; + /** Re-render the page after a success. Off only for Test connection, which changes nothing. */ + refresh?: boolean; +} + +/** + * Gate, limit, parse, act, refresh — each only as far as the last one earned. + * + * The gate runs before the parse, as on every admin action: an anonymous + * prodder learns nothing about the input shape for free. A refusal refreshes + * nothing, unless it changed something anyway — `createDatabases` stopping + * part-way has still made databases and written them into the mapping, and + * the page must show them. + */ +async function run( + options: RunOptions

, + act: (identity: OwnerIdentity, parsed: P) => Promise +): Promise { + const gate = await authorizeAdminAction("mirror.manage"); + if (!gate.ok) return gate; + const { identity } = gate; + const userId = identity.userId; + if (!userId) return { ok: false, error: "not_signed_in" }; + + if (options.setup) { + const { allowed } = await rateLimitAsync(`mirror-setup:${identity.rateLimitKey}`, MIRROR_SETUP_TIER); + if (!allowed) return { ok: false, error: "rate_limited" }; + } + + const parsed = options.schema.safeParse(options.input); + if (!parsed.success) return { ok: false, error: "invalid_field" }; + + let result: T | MirrorActionFailure; + try { + result = await act({ ...identity, userId }, parsed.data); + } catch (error) { + console.error(`[${SURFACE}] the action failed: ${describe(error, options.secret)}`); + return { ok: false, error: "failed" }; + } + + const changedAnyway = !result.ok && ((result as MirrorActionFailure).created?.length ?? 0) > 0; + if ((result.ok && options.refresh !== false) || changedAnyway) revalidatePath(MIRROR_PATH); + return result; +} diff --git a/v5/src/app/admin/mirror/page.tsx b/v5/src/app/admin/mirror/page.tsx new file mode 100644 index 0000000..061013f --- /dev/null +++ b/v5/src/app/admin/mirror/page.tsx @@ -0,0 +1,167 @@ +import { getTranslations } from "next-intl/server"; +import { AdminNotice } from "../../../components/admin/AdminNotice"; +import { MirrorConnect } from "../../../components/admin/MirrorConnect"; +import { MirrorControls } from "../../../components/admin/MirrorControls"; +import { MirrorMapping } from "../../../components/admin/MirrorMapping"; +import { MirrorStatus } from "../../../components/admin/MirrorStatus"; +import { resolveIdentityFromHeaders } from "../../../lib/auth/identity"; +import { can } from "../../../lib/auth/permissions"; +import { getMirrorViewForOwner } from "../../../lib/data/mirrors"; +import { labTimezone } from "../../../lib/lab-time"; +import { mirrorKeyAvailable } from "../../../lib/mirror/token-crypto"; +import type { MirrorView } from "../../../lib/mirror/types"; +import { siteConfig } from "../../../lib/site-config"; +import { + connect, + createDatabases, + disconnect, + saveMapping, + setPaused, + syncNow, + testConnection, +} from "./actions"; +import type { MirrorActions } from "./action-result"; + +/** + * `/admin/mirror` — the signed-in admin's own Notion mirror (spec §3.8, §5.8, + * §6, §8). + * + * Requires `mirror.manage`. The layout above let anyone holding an admin + * permission through; the exact refusal happens here and is *said*, never a + * 404. + * + * **Owner-only by construction.** The page reads one row — + * `getMirrorViewForOwner(identity.userId)` — so it cannot show anybody else's + * mirror, and the actions it hands down take no mirror id at all (§8). When + * Niti opens this page she sees her own mirror, or the form to make one; + * Isaac's is not hers to see (§3.8 "Owners"). + * + * **No Notion call happens during render.** Status is what the last push + * recorded; the page is a settings page over a database row, and Notion being + * down changes nothing about whether it renders (§5.8). + * + * The states, in the order they are decided: + * + * - **No `AUTH_SECRET`** — a token cannot be encrypted, so there is no form to + * fill; the page says why instead (§8 "Secrets at rest"). + * - **No mirror yet** — the §4.14 steps and `MirrorConnect`. + * - **A mirror whose token is gone or refused** — disconnected, or its last + * push failed `unauthorized` / `token_unreadable` (§5.8): "the connection + * needs a new token", `MirrorConnect` again, and the mapping shown and kept + * so reconnecting duplicates nothing. + * - **Connected** — `MirrorStatus`, `MirrorControls`, `MirrorMapping`. + * + * **A database that cannot be reached is said, not papered over** (§6 States). + */ + +export const metadata = { + title: `Notion mirror — ${siteConfig.name}`, +}; + +/** The last push failed because of the token itself; a new one is the fix. */ +const TOKEN_ERRORS: ReadonlySet = new Set(["unauthorized", "token_unreadable"]); + +export default async function AdminMirrorPage() { + const t = await getTranslations("admin"); + const identity = await resolveIdentityFromHeaders(); + + if (!can(identity, "mirror.manage") || !identity.userId) return ; + + let view: MirrorView | null; + try { + view = await getMirrorViewForOwner(identity.userId); + } catch (err) { + console.error("[admin/mirror] could not read the mirror", err); + return ( + +

+ {t("mirror.unavailable")} +

+ + ); + } + + const actions: MirrorActions = { + testConnection, + connect, + createDatabases, + saveMapping, + syncNow, + setPaused, + disconnect, + }; + const timeZone = labTimezone(); + + if (!mirrorKeyAvailable()) { + return ( + +
+

{t("mirror.keyUnavailableTitle")}

+

{t("mirror.keyUnavailableBody")}

+
+ {view ? : null} + + ); + } + + if (!view) { + return ( + +
+

{t("mirror.howToTitle")}

+
    +
  1. {t("mirror.howToIntegration")}
    2. +
  2. {t("mirror.howToPage")}
    3. +
  3. {t("mirror.howToShare")}
    4. +
  4. {t("mirror.howToPaste")}
    5. +
+

{t("mirror.privacyNote")}

+
+ + + ); + } + + const needsToken = !view.connected || (view.lastError !== null && TOKEN_ERRORS.has(view.lastError.code)); + + if (needsToken) { + return ( + +
+

{t("mirror.needsTokenTitle")}

+

{t(view.connected ? "mirror.needsTokenRejected" : "mirror.needsTokenDisconnected")}

+
+ + + {view.connected ? : null} + + + ); + } + + return ( + + + + +

{t("mirror.privacyNote")}

+ + ); +} + +/** The page's header, shared by every state. */ +async function MirrorSection({ children }: { children: React.ReactNode }) { + const t = await getTranslations("admin"); + return ( +
+
+

{t("eyebrow")}

+

{t("mirrorTitle")}

+ {/* No placeholder in this string: `/admin/page.tsx` renders the same + key without arguments (Article 6). */} +

{t("mirrorLede")}

+
+ {children} +
+ ); +} diff --git a/v5/src/app/admin/page.tsx b/v5/src/app/admin/page.tsx index 528292c..5db8044 100644 --- a/v5/src/app/admin/page.tsx +++ b/v5/src/app/admin/page.tsx @@ -7,12 +7,14 @@ import { can, type Permission } from "../../lib/auth/permissions"; * `/admin` — the index the header's `AdminLink` points at. * * This is still a short list rather than the `AdminHome` of spec §6 (counts, - * open tickets, mirror status). The intake queue now exists — `/admin/intake`, - * Phase 6 — and is listed here like every other surface; the counts and the - * mirror's status still wait for the phases that add them. What the page must - * do is be honest: it lists exactly the surfaces the viewer's own permissions - * open, so nobody follows a link into a refusal, and a SuperMaker who holds - * `tools.edit` but not `users.manage` sees the inventory and not the roster. + * open tickets, mirror status). Every surface the spec names now exists — the + * Notion mirror, `/admin/mirror`, arrived last, in Phase 8 — and each is listed + * here from one table; the counts and the mirror's status on this page are + * still `AdminHome`'s, not built. What the page must do is be honest: it lists + * exactly the surfaces the viewer's own permissions open, so nobody follows a + * link into a refusal, and a SuperMaker who holds `tools.edit` but not + * `users.manage` sees the inventory and not the roster. The mirror entry opens + * the viewer's *own* mirror — there is no page listing anybody else's (§8). * * The layout above has already established that this person may see an admin * surface at all. @@ -36,6 +38,7 @@ const SURFACES: ReadonlyArray<{ href: string; permission: Permission; key: strin { href: "/admin/corrections", permission: "feedback.manage", key: "corrections" }, { href: "/admin/projects", permission: "projects.moderate", key: "projects" }, { href: "/admin/users", permission: "users.manage", key: "users" }, + { href: "/admin/mirror", permission: "mirror.manage", key: "mirror" }, ]; export default async function AdminHomePage() { @@ -60,8 +63,6 @@ export default async function AdminHomePage() { ) : (

{t("indexNothingYet")}

)} - -

{t("indexMoreComing")}

); } diff --git a/v5/src/app/admin/projects/actions.mirror.test.ts b/v5/src/app/admin/projects/actions.mirror.test.ts new file mode 100644 index 0000000..9adfd66 --- /dev/null +++ b/v5/src/app/admin/projects/actions.mirror.test.ts @@ -0,0 +1,81 @@ +// @vitest-environment node +import { nextCacheMock } from "../../../../test/mocks/next-cache"; +import { nextHeadersMock, setMockHeaders } from "../../../../test/mocks/next-headers"; + +vi.mock("next/cache", () => nextCacheMock()); +vi.mock("next/headers", () => nextHeadersMock()); + +// The trigger has its own tests; here it is only asked whether it was called. +const mirror = vi.hoisted(() => ({ requestMirrorPush: vi.fn() })); + +vi.mock("../../../lib/mirror/trigger", () => ({ requestMirrorPush: mirror.requestMirrorPush })); + +import { resetAuthForTests } from "../../../lib/auth/config"; +import { getDb, resetDbForTests } from "../../../lib/db/client"; +import { auditEvents, projects, session, user } from "../../../lib/db/schema/index"; +import { signInAsNew } from "../../../../test/utils/session"; +import { setPublished } from "./actions"; + +/** + * Publishing and unpublishing a project are mirror triggers (spec §3.8 + * trigger 1): the mirror carries published projects only, so both directions + * change what it should hold. A refused press changes nothing and asks for + * nothing. + */ + +const AUTH_SECRET = "admin-projects-mirror-test-secret"; + +let projectId: string; + +beforeEach(async () => { + vi.stubEnv("DATABASE_URL", ""); + vi.stubEnv("AUTH_SECRET", AUTH_SECRET); + vi.stubEnv("AUTH_SUPER_ADMIN_EMAILS", ""); + resetAuthForTests(); + mirror.requestMirrorPush.mockReset().mockResolvedValue(undefined); + + const db = await getDb(); + await db.delete(auditEvents); + await db.delete(projects); + await db.delete(session); + await db.delete(user); + const [row] = await db + .insert(projects) + .values({ slug: "resin-dice-tower", title: "Resin dice tower", body: "A dice tower.", published: false }) + .returning({ id: projects.id }); + projectId = row.id; +}); + +afterEach(() => { + resetAuthForTests(); + resetDbForTests(); +}); + +it("requests a push after publishing, and again after unpublishing", async () => { + const admin = await signInAsNew({ email: "niti@cornell.edu", role: "admin" }); + setMockHeaders({ cookie: admin.cookie }); + + expect(await setPublished({ projectId, published: true })).toEqual({ ok: true }); + expect(mirror.requestMirrorPush).toHaveBeenCalledTimes(1); + + expect(await setPublished({ projectId, published: false })).toEqual({ ok: true }); + expect(mirror.requestMirrorPush).toHaveBeenCalledTimes(2); +}); + +it("requests nothing when the caller is refused", async () => { + const student = await signInAsNew({ email: "casey@cornell.edu", role: "user" }); + setMockHeaders({ cookie: student.cookie }); + + expect(await setPublished({ projectId, published: true })).toEqual({ ok: false, error: "not_permitted" }); + expect(mirror.requestMirrorPush).not.toHaveBeenCalled(); +}); + +it("requests nothing when the project does not exist", async () => { + const admin = await signInAsNew({ email: "niti@cornell.edu", role: "admin" }); + setMockHeaders({ cookie: admin.cookie }); + + const result = await setPublished({ projectId: crypto.randomUUID(), published: true }); + + expect(result.ok).toBe(false); + expect(mirror.requestMirrorPush).not.toHaveBeenCalled(); +}); diff --git a/v5/src/app/admin/projects/actions.ts b/v5/src/app/admin/projects/actions.ts index 7c1f0d4..8869fd7 100644 --- a/v5/src/app/admin/projects/actions.ts +++ b/v5/src/app/admin/projects/actions.ts @@ -3,6 +3,7 @@ import { record } from "../../../lib/admin/audit-warning"; import { runQueueWrite } from "../../../lib/admin/queue-write"; import { setProjectPublished } from "../../../lib/data/projects"; +import { requestMirrorPush } from "../../../lib/mirror/trigger"; import { invalidateProjects } from "../../../lib/revalidate"; import { ADMIN_PROJECTS_PATH, type ProjectActionResult } from "./action-result"; @@ -27,6 +28,10 @@ import { ADMIN_PROJECTS_PATH, type ProjectActionResult } from "./action-result"; * - **It checks `projects.moderate`**, which is its own permission and not * `tools.publish`. Publishing a machine and publishing somebody's write-up * are different jobs, and the declaration already says so. + * - **It tells the Notion mirror.** The mirror carries published projects only + * (§3.8), so both directions change what it should hold, and + * `requestMirrorPush()` runs after the invalidation (§3.8 trigger 1). It + * never throws; a refused write never reaches it. */ /** Names this surface in the console line a missing audit event leaves behind. */ @@ -66,6 +71,7 @@ export async function setPublished(input: { // write has nothing to show, and busting the gallery for it would cost a // full re-read for free. invalidateProjects(); + await requestMirrorPush(); return recorded ? undefined : "audit_unavailable"; }, diff --git a/v5/src/app/api/chat/route.test.ts b/v5/src/app/api/chat/route.test.ts index ede6144..d33044f 100644 --- a/v5/src/app/api/chat/route.test.ts +++ b/v5/src/app/api/chat/route.test.ts @@ -75,7 +75,9 @@ vi.mock("next/cache", () => ({ })); import { eq, inArray } from "drizzle-orm"; +import { http, HttpResponse } from "msw"; import { POST } from "@/app/api/chat/route"; +import { manualSourceKey } from "@/lib/data/manual-archives"; import { getDb, resetDbForTests } from "@/lib/db/client"; import { attachments, @@ -85,6 +87,7 @@ import { units, } from "@/lib/db/schema/index"; import { resetAuthForTests } from "@/lib/auth/config"; +import { server } from "../../../../test/msw/server"; import { signInAsNew } from "../../../../test/utils/session"; /** @@ -578,6 +581,48 @@ describe("PDF manual collection (focused tool)", () => { expect(captured.args.system).toContain("Scanned manual"); }); + it("attaches the archived copy instead of the manufacturer's link, once, and marks it attached", async () => { + const SOURCE = "https://maker.test/support/form-4-manual"; + const ARCHIVE = "https://blob.test/manuals/form-4/manual-abc.pdf"; + await addResources([{ title: "Form 4 manual", url: SOURCE }]); + const id = insertedResourceIds[insertedResourceIds.length - 1]; + const db = await getDb(); + await db.insert(attachments).values({ + ownerType: "resource", + ownerId: id, + blobPathname: "manuals/form-4/manual-abc.pdf", + access: "public", + publicUrl: ARCHIVE, + contentType: "application/pdf", + sourceKey: manualSourceKey(id, SOURCE), + }); + + const fetched: string[] = []; + server.use( + http.get(ARCHIVE, ({ request }) => { + fetched.push(request.url); + return HttpResponse.arrayBuffer(new Uint8Array([0x25, 0x50, 0x44, 0x46]).buffer, { + headers: { "content-type": "application/pdf" }, + }); + }) + // No handler for SOURCE: fetching it would fail the test. + ); + + await POST(chatRequest({ messages: [userMessage("help")], toolId: "form-4" })); + + expect(fetched).toEqual([ARCHIVE]); + const firstUser = captured.args.messages.find((m: any) => m.role === "user"); + expect((firstUser.content as any[]).filter((p) => p.type === "file")).toHaveLength(1); + const system: string = captured.args.system; + expect(system).toContain(`${ARCHIVE} (attached)`); + // One line for the one manual in the annotated list — the copy and the + // source are not listed as two resources. + const annotated = system.slice(system.indexOf("## Attached manuals vs. fetchable resources")); + expect(annotated.split("\n").filter((line) => line.includes("Form 4 manual"))).toEqual([ + `- [Manual] Form 4 manual — ${ARCHIVE} (attached)`, + ]); + }); + it("does not run manual collection when no toolId is provided", async () => { await addResources([{ title: "Manual 1", url: "https://x.test/m1.pdf" }]); const fetchMock = vi.fn(async () => okPdf(PDF_SMALL)); diff --git a/v5/src/app/api/chat/route.ts b/v5/src/app/api/chat/route.ts index 4ca2ca4..b7caabc 100644 --- a/v5/src/app/api/chat/route.ts +++ b/v5/src/app/api/chat/route.ts @@ -42,6 +42,12 @@ const PDF_FETCH_UA = "Mozilla/5.0 (compatible; MakerLabBot/1.0)"; interface AttachedManual { title: string; url: string; + /** + * The resource's own link when `url` is its archived copy. The tool page's + * links come from a cached read that may predate the copy, so "(attached)" + * matches either. + */ + sourceUrl?: string; /** Base64-encoded PDF bytes, present only if the server-side fetch succeeded. */ data: string; } @@ -72,7 +78,7 @@ export async function POST(req: Request) { ? await collectToolManuals(focused.id) : { manuals: [], skipped: 0 }; if (focused) { - const hosts = uniqueHosts(focused.links.map((l) => l.href)); + const hosts = uniqueHosts(linkUrls(focused)); console.info( `[chat] focused tool: ${focused.name} (${focused.id}), links: ${focused.links.length}` ); @@ -148,7 +154,7 @@ export async function POST(req: Request) { citations: { enabled: true }, ...(focused ? (() => { - const hosts = uniqueHosts(focused.links.map((l) => l.href)); + const hosts = uniqueHosts(linkUrls(focused)); return hosts.length ? { allowedDomains: hosts } : {}; })() : {}), @@ -371,6 +377,14 @@ function parsePhotoHints(text: string): PhotoHint[] { // ── Helpers (focused tool / manuals) ─────────────────────────────── +/** + * Every URL the focused tool's links name — the archived copy *and* the + * manufacturer's original, so `web_fetch` may still fall back to the source. + */ +function linkUrls(tool: MakerLabTool): string[] { + return tool.links.flatMap((link) => (link.sourceHref ? [link.href, link.sourceHref] : [link.href])); +} + function uniqueHosts(urls: string[]): string[] { const set = new Set(); for (const u of urls) { @@ -389,7 +403,15 @@ function isPdfUrl(url: string | null | undefined): boolean { return cleaned.endsWith(".pdf"); } +/** + * The PDF to attach for one resource: its archived copy in Blob first (the + * manual archive — it outlives the manufacturer's link, and matches the href + * the tool's links carry, which keeps "(attached)" honest), then the source + * link, then an uploaded file. One per resource, so a manual is never attached + * twice. + */ function pickPdfUrl(resource: ToolResource): string | null { + if (resource.archivedUrl) return resource.archivedUrl; if (isPdfUrl(resource.url)) return resource.url; return resource.fileUrls.find(isPdfUrl) ?? null; } @@ -471,7 +493,9 @@ async function collectToolManuals( skipped += 1; continue; } - manuals.push({ title, url, data }); + manuals.push( + r.archivedUrl && r.url && url === r.archivedUrl ? { title, url, sourceUrl: r.url, data } : { title, url, data } + ); } } catch (err) { // Never let base64 collection take down the request; fall back to web_fetch. @@ -542,7 +566,7 @@ function appendManualSections( ); if (focused && focused.links.length > 0) { - const attachedUrls = new Set(manuals.map((m) => m.url)); + const attachedUrls = new Set(manuals.flatMap((m) => (m.sourceUrl ? [m.url, m.sourceUrl] : [m.url]))); const annotated = focused.links .map((link) => { const tag = attachedUrls.has(link.href) ? " (attached)" : ""; diff --git a/v5/src/app/api/cron/daily/route.test.ts b/v5/src/app/api/cron/daily/route.test.ts index a9052fb..1465a74 100644 --- a/v5/src/app/api/cron/daily/route.test.ts +++ b/v5/src/app/api/cron/daily/route.test.ts @@ -44,10 +44,42 @@ vi.mock("../../../../lib/cron/pending-expiry", async (importOriginal) => { }; }); -import { sql } from "drizzle-orm"; +/** + * The mirror stage (Phase 8). Starting a workflow is mocked at `start.ts` — + * the workflow tier has its own test — so the stage runs its real query + * against the seeded database and hands `startMirrorPush` whatever is due. + * `throwOnce` fails the whole stage, as `pendingExpiryOverride` does above. + */ +const mirrorStage = vi.hoisted(() => ({ throwOnce: null as Error | null, startMirrorPush: vi.fn() })); + +vi.mock("../../../../lib/mirror/start", () => ({ startMirrorPush: mirrorStage.startMirrorPush })); + +vi.mock("../../../../lib/cron/mirror-backstop", async (importOriginal) => { + const actual = + await importOriginal(); + return { + ...actual, + runMirrorBackstop: (...args: Parameters) => { + if (mirrorStage.throwOnce) { + const error = mirrorStage.throwOnce; + mirrorStage.throwOnce = null; + return Promise.reject(error); + } + return actual.runMirrorBackstop(...args); + }, + }; +}); + +/** The manual archive backfill: starting its workflow is mocked at `start.ts`. */ +const manualStage = vi.hoisted(() => ({ startManualArchive: vi.fn() })); + +vi.mock("../../../../lib/manuals/start", () => ({ startManualArchive: manualStage.startManualArchive })); + +import { eq, sql } from "drizzle-orm"; +import { saveMirrorConnection } from "@/lib/data/mirrors"; import { getDb, resetDbForTests } from "@/lib/db/client"; import { DEMO_ACCOUNTS } from "@/lib/db/demo-seed"; -import { attachments, pendingTools, tools } from "@/lib/db/schema/index"; +import { attachments, notionMirrors, pendingTools, resources, tools } from "@/lib/db/schema/index"; import { GET } from "./route"; /** @@ -66,6 +98,9 @@ beforeEach(async () => { vi.stubEnv("BLOB_READ_WRITE_TOKEN", "vercel_blob_rw_test"); pendingExpiryOverride.throwOnce = null; + mirrorStage.throwOnce = null; + mirrorStage.startMirrorPush.mockReset().mockResolvedValue({ ok: true, runId: "run-1" }); + manualStage.startManualArchive.mockReset().mockResolvedValue(true); blob.configured.value = true; blob.put.mockReset().mockResolvedValue({ pathname: "written" }); blob.list.mockReset().mockResolvedValue([]); @@ -74,6 +109,7 @@ beforeEach(async () => { const db = await getDb(); await db.delete(pendingTools); await db.delete(attachments); + await db.delete(notionMirrors); }); afterAll(() => { @@ -318,6 +354,119 @@ describe("GET /api/cron/daily — the cleanup stage", () => { }); }); +describe("GET /api/cron/daily — the mirror stage", () => { + /** An active, never-synced mirror owned by the demo admin: due tonight. */ + async function activeMirror(): Promise { + const { mirror } = await saveMirrorConnection({ + ownerUserId: DEMO_ACCOUNTS.admin.id, + tokenCiphertext: new Uint8Array([1, 2, 3]), + parentPageId: "0f5e4a3c-1111-2222-3333-444455556666", + parentPageTitle: null, + }); + return mirror.id; + } + + it("reports the mirror stage, and starts nothing when there is no mirror", async () => { + const res = await GET(authorized()); + const body = await res.json(); + + expect(res.status).toBe(200); + expect(body.mirror).toEqual({ due: 0, started: 0, failed: 0 }); + expect(mirrorStage.startMirrorPush).not.toHaveBeenCalled(); + }); + + it("starts a push for a mirror that has never synced", async () => { + vi.spyOn(console, "info").mockImplementation(() => {}); + const id = await activeMirror(); + + const body = await (await GET(authorized())).json(); + + expect(body.ok).toBe(true); + expect(body.mirror).toEqual({ due: 1, started: 1, failed: 0 }); + expect(mirrorStage.startMirrorPush).toHaveBeenCalledWith(id); + }); + + it("returns 500 with stage 'mirror' when the stage throws, and still reports every earlier stage", async () => { + mirrorStage.throwOnce = new Error("mirror backstop blew up"); + + const res = await GET(authorized()); + const body = await res.json(); + + expect(res.status).toBe(500); + expect(body.stage).toBe("mirror"); + expect(body.error).toContain("mirror backstop blew up"); + expect(body.backup.pathname).toMatch(/^backups\//); + expect(body.pendingExpiry).toBeDefined(); + expect(body.cleanup).toBeDefined(); + }); + + it("returns 500 with stage 'mirror' when a push could not be started — never a quiet 200", async () => { + vi.spyOn(console, "info").mockImplementation(() => {}); + await activeMirror(); + mirrorStage.startMirrorPush.mockResolvedValue({ ok: false }); + + const res = await GET(authorized()); + const body = await res.json(); + + expect(res.status).toBe(500); + expect(body.stage).toBe("mirror"); + expect(body.mirror).toEqual({ due: 1, started: 0, failed: 1 }); + expect(body.cleanup).toBeDefined(); + }); +}); + +describe("GET /api/cron/daily — the manual archive stage", () => { + const TITLE = "Cron test manual"; + + async function manual(): Promise { + const db = await getDb(); + const [tool] = await db.select({ id: tools.id }).from(tools).limit(1); + const [row] = await db + .insert(resources) + .values({ toolId: tool.id, title: TITLE, type: "Manual", url: "https://maker.test/cron-manual.pdf" }) + .returning({ id: resources.id }); + return row.id; + } + + afterEach(async () => { + const db = await getDb(); + await db.delete(resources).where(eq(resources.title, TITLE)); + }); + + it("reports zeros and starts nothing when no manual is due", async () => { + const body = await (await GET(authorized())).json(); + + expect(body.ok).toBe(true); + expect(body.manuals).toEqual({ due: 0, queued: 0, failed: 0 }); + expect(manualStage.startManualArchive).not.toHaveBeenCalled(); + }); + + it("hands a due manual to an archive run and reports the counts", async () => { + vi.spyOn(console, "info").mockImplementation(() => {}); + const id = await manual(); + + const body = await (await GET(authorized())).json(); + + expect(body.ok).toBe(true); + expect(body.manuals).toEqual({ due: 1, queued: 1, failed: 0 }); + expect(manualStage.startManualArchive).toHaveBeenCalledWith([id]); + }); + + it("returns 500 with stage 'manuals' when the run could not be started", async () => { + vi.spyOn(console, "info").mockImplementation(() => {}); + await manual(); + manualStage.startManualArchive.mockResolvedValue(false); + + const res = await GET(authorized()); + const body = await res.json(); + + expect(res.status).toBe(500); + expect(body.stage).toBe("manuals"); + expect(body.manuals).toEqual({ due: 1, queued: 0, failed: 1 }); + expect(body.mirror).toBeDefined(); + }); +}); + describe("GET /api/cron/daily — blob not configured", () => { it("refuses with 503 naming the variable rather than running half the job", async () => { blob.configured.value = false; diff --git a/v5/src/app/api/cron/daily/route.ts b/v5/src/app/api/cron/daily/route.ts index d37ff7f..ce73a37 100644 --- a/v5/src/app/api/cron/daily/route.ts +++ b/v5/src/app/api/cron/daily/route.ts @@ -1,6 +1,8 @@ import { getBlobStore, isBlobConfigured } from "../../../../lib/blob"; import { runBackup } from "../../../../lib/cron/backup"; import { runCleanup } from "../../../../lib/cron/cleanup"; +import { runManualArchiveBackfill } from "../../../../lib/cron/manual-archive"; +import { runMirrorBackstop } from "../../../../lib/cron/mirror-backstop"; import { runPendingExpiry } from "../../../../lib/cron/pending-expiry"; import { rateLimitAsync } from "../../../../lib/rate-limit"; import { resolveIdentity } from "../../../../lib/auth/identity"; @@ -25,8 +27,17 @@ import { resolveIdentity } from "../../../../lib/auth/identity"; * `identified` for two weeks, always well past the 24-hour orphan window — * so a photo an expired item held is deleted from Blob and from the table * in this same run (§4.10 "its attachments deleted"). - * - * Mirror pushes (§3.8) join this list in a later phase; it has no writer yet. + * 4. **Mirror backstop** (Phase 8) — a `mirrorPush` workflow started for every + * active Notion mirror whose data is newer than its last sync, or whose + * last push was not `ok` (§3.8 trigger 3). The stage only starts the runs; + * each pushes in its own workflow, outside this function's 60 seconds. A + * run that could not be started fails the stage, as a throw does. + * 5. **Manual archive backfill** — up to ten Manual resources whose link has + * no PDF copy in Blob yet, handed to one `archiveManuals` run + * (`src/lib/cron/manual-archive.ts`). Backfills imported manuals over time + * and catches any approval whose run never started. Like the mirror stage + * it only starts the run, and a run that could not be started fails the + * stage. * * **Nothing here fails quietly.** Every stage reports, and any one failing * makes the whole invocation non-200 so it shows in Vercel's cron log as @@ -131,9 +142,9 @@ export async function GET(req: Request) { ); } + let cleanup: Awaited>; try { - const cleanup = await runCleanup(store); - return Response.json({ ok: true, backup, pendingExpiry, cleanup }); + cleanup = await runCleanup(store); } catch (error) { console.error("[cron] cleanup failed:", error); return Response.json( @@ -141,4 +152,64 @@ export async function GET(req: Request) { { status: 500 } ); } + + // Last, because it is the least urgent and the only stage that hands work + // to something else: every earlier stage has landed, and reports, whatever + // happens here. + let mirror: Awaited>; + try { + mirror = await runMirrorBackstop(); + } catch (error) { + console.error("[cron] mirror backstop failed:", error); + return Response.json( + { ok: false, stage: "mirror", backup, pendingExpiry, cleanup, error: message(error) }, + { status: 500 } + ); + } + if (mirror.failed > 0) { + // The ids and the reasons are already in the log (`start.ts`); the body + // says how many, so the cron log shows a failed invocation. + return Response.json( + { + ok: false, + stage: "mirror", + backup, + pendingExpiry, + cleanup, + mirror, + error: `${mirror.failed} mirror push(es) could not be started`, + }, + { status: 500 } + ); + } + + // After the mirror, for the same reason it is after everything else: it only + // hands work to a workflow, and every earlier stage has already reported. + let manuals: Awaited>; + try { + manuals = await runManualArchiveBackfill(); + } catch (error) { + console.error("[cron] manual archive backfill failed:", error); + return Response.json( + { ok: false, stage: "manuals", backup, pendingExpiry, cleanup, mirror, error: message(error) }, + { status: 500 } + ); + } + if (manuals.failed > 0) { + return Response.json( + { + ok: false, + stage: "manuals", + backup, + pendingExpiry, + cleanup, + mirror, + manuals, + error: "the manual archive run could not be started", + }, + { status: 500 } + ); + } + + return Response.json({ ok: true, backup, pendingExpiry, cleanup, mirror, manuals }); } diff --git a/v5/src/app/api/dev-blob/[...path]/route.test.ts b/v5/src/app/api/dev-blob/[...path]/route.test.ts new file mode 100644 index 0000000..c92d9d2 --- /dev/null +++ b/v5/src/app/api/dev-blob/[...path]/route.test.ts @@ -0,0 +1,80 @@ +// @vitest-environment node +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { createLocalBlobBackend } from "../../../../lib/blob-local"; +import { GET } from "./route"; + +/** `GET /api/dev-blob/…` against a temporary `.blob-data/`. */ + +let dir: string; + +beforeEach(async () => { + dir = await mkdtemp(join(tmpdir(), "dev-blob-route-")); + vi.spyOn(process, "cwd").mockReturnValue(dir); + vi.stubEnv("BLOB_READ_WRITE_TOKEN", ""); + vi.stubEnv("VERCEL", ""); + vi.stubEnv("NODE_ENV", "development"); + vi.stubEnv("BLOB_LOCAL_DISABLE", ""); +}); + +afterEach(async () => { + await rm(dir, { recursive: true, force: true }); +}); + +function get(pathname: string) { + return GET(new Request(`http://localhost:3001/api/dev-blob/${pathname}`), { + params: Promise.resolve({ path: pathname.split("/") }), + }); +} + +describe("GET /api/dev-blob/[...path]", () => { + it("serves a public file with its stored content type", async () => { + await createLocalBlobBackend().put("uploads/tool/plate.png", new Uint8Array([9, 8, 7]), { + access: "public", + contentType: "image/png", + }); + + const res = await get("uploads/tool/plate.png"); + expect(res.status).toBe(200); + expect(res.headers.get("content-type")).toBe("image/png"); + expect(res.headers.get("x-content-type-options")).toBe("nosniff"); + expect([...new Uint8Array(await res.arrayBuffer())]).toEqual([9, 8, 7]); + }); + + it("404s a private file", async () => { + await createLocalBlobBackend().put("backups/2026-07-29.json", "{}", { + access: "private", + contentType: "application/json", + }); + expect((await get("backups/2026-07-29.json")).status).toBe(404); + }); + + it("404s a missing file", async () => { + expect((await get("uploads/nope.png")).status).toBe(404); + }); + + it("404s traversal and the metadata folder", async () => { + await createLocalBlobBackend().put("uploads/a.png", "x", { access: "public", contentType: "image/png" }); + expect((await get("../package.json")).status).toBe(404); + expect((await get("uploads/../../etc/passwd")).status).toBe(404); + expect((await get(".meta/uploads/a.png.json")).status).toBe(404); + }); + + it("404s everything outside local mode, even a public file", async () => { + await createLocalBlobBackend().put("uploads/a.png", "x", { access: "public", contentType: "image/png" }); + expect((await get("uploads/a.png")).status).toBe(200); + + vi.stubEnv("VERCEL", "1"); + expect((await get("uploads/a.png")).status).toBe(404); + vi.stubEnv("VERCEL", ""); + vi.stubEnv("NODE_ENV", "production"); + expect((await get("uploads/a.png")).status).toBe(404); + vi.stubEnv("NODE_ENV", "development"); + vi.stubEnv("BLOB_READ_WRITE_TOKEN", "vercel_blob_rw_test"); + expect((await get("uploads/a.png")).status).toBe(404); + vi.stubEnv("BLOB_READ_WRITE_TOKEN", ""); + vi.stubEnv("BLOB_LOCAL_DISABLE", "1"); + expect((await get("uploads/a.png")).status).toBe(404); + }); +}); diff --git a/v5/src/app/api/dev-blob/[...path]/route.ts b/v5/src/app/api/dev-blob/[...path]/route.ts new file mode 100644 index 0000000..6b29988 --- /dev/null +++ b/v5/src/app/api/dev-blob/[...path]/route.ts @@ -0,0 +1,54 @@ +import { createLocalBlobBackend } from "../../../../lib/blob-local"; +import { blobMode } from "../../../../lib/blob-mode"; + +/** + * `GET /api/dev-blob/` — the local Blob store's public URL. + * + * In local development with no `BLOB_READ_WRITE_TOKEN`, uploads, promoted + * photos, archived manuals and backups are written to `.blob-data/` + * (`lib/blob-local.ts`), and a public file's URL points here. This serves it + * with the content type it was stored with, as a public Vercel Blob URL would. + * + * - **Inert outside local mode.** On Vercel, in a production build, with a + * real token, or with `BLOB_LOCAL_DISABLE=1`, every request is a 404 — the + * route exists in production and does nothing there. + * - **Private files are 404**, exactly as a private blob URL is unreachable + * without the token. Code that needs a private file's bytes reads them + * through the store, not a URL. + * - **No path traversal.** The pathname is validated before it touches the + * disk (`resolveLocalPath`); anything that could leave the folder or reach + * its metadata is a 404. + */ + +// `runtime` cannot be set when nextConfig.cacheComponents is enabled. +// Default Node.js runtime is used. + +const notFound = () => new Response("Not found", { status: 404 }); + +export async function GET( + _req: Request, + { params }: { params: Promise<{ path: string[] }> } +): Promise { + if (blobMode() !== "local") return notFound(); + + const { path } = await params; + if (!Array.isArray(path) || path.length === 0) return notFound(); + + let blob: Awaited["read"]>>; + try { + blob = await createLocalBlobBackend().read(path.join("/")); + } catch { + return notFound(); + } + if (!blob || blob.meta.access !== "public") return notFound(); + + return new Response(Buffer.from(blob.body), { + status: 200, + headers: { + "Content-Type": blob.meta.contentType, + "Content-Length": String(blob.body.byteLength), + "Cache-Control": "no-store", + "X-Content-Type-Options": "nosniff", + }, + }); +} diff --git a/v5/src/components/DetailShell.tsx b/v5/src/components/DetailShell.tsx index 8b7c2c3..b0a86c6 100644 --- a/v5/src/components/DetailShell.tsx +++ b/v5/src/components/DetailShell.tsx @@ -1,6 +1,8 @@ import Image from "next/image"; import Link from "next/link"; import { useTranslations } from "next-intl"; +import ReactMarkdown from "react-markdown"; +import remarkGfm from "remark-gfm"; import type { MakerLabProject, MakerLabTool, ToolStatus } from "./catalog-types"; interface DetailShellProps { @@ -85,7 +87,10 @@ export function DetailShell({ tool, projects = [] }: DetailShellProps) {

{tool.name}

-

{tool.description}

+ {/* Descriptions are Markdown (research folds specs in as a list); no raw HTML, as for projects. */} +
+ {tool.description} +
diff --git a/v5/src/components/IntakeTableCard.tsx b/v5/src/components/IntakeTableCard.tsx index 1d89a63..da2f3f4 100644 --- a/v5/src/components/IntakeTableCard.tsx +++ b/v5/src/components/IntakeTableCard.tsx @@ -253,6 +253,9 @@ export function IntakeTableCard({ payload }: IntakeTableCardProps) { disabled={locked || eligibleIds.length === 0} onChange={toggleAll} /> + {t("table.columnPhoto")} {t("table.columnName")} @@ -437,6 +440,9 @@ function IntakeRow({ row, t, selected, locked, saving, onToggle, onEditingChange disabled={disabled || unresolved} onChange={onToggle} /> + diff --git a/v5/src/components/admin/MirrorConnect.test.tsx b/v5/src/components/admin/MirrorConnect.test.tsx new file mode 100644 index 0000000..4fa58a9 --- /dev/null +++ b/v5/src/components/admin/MirrorConnect.test.tsx @@ -0,0 +1,144 @@ +import { render, screen, userEvent } from "../../../test/utils/render"; +import type { MirrorConnectActionResult, MirrorTestResult } from "../../app/admin/mirror/action-result"; +import { MirrorConnect, type MirrorConnectActions } from "./MirrorConnect"; + +/** + * The Connect form (spec §3.8 "Connect", §8). + * + * The token field is a password field the browser will not autofill; Test + * connection shows the page's title and nothing else; Connect clears the token + * the moment it lands; and a refusal is a sentence, never a success. + */ + +const TOKEN = "ntn_COMPONENTtestToken0123456789"; +const PAGE_URL = "https://www.notion.so/acme/MakerLab-Tools-mirror-0f5e4a3c111122223333444455556666"; + +function actions(over: Partial = {}): MirrorConnectActions { + return { + testConnection: vi.fn( + async (): Promise => ({ + ok: true, + pageId: "0f5e4a3c-1111-2222-3333-444455556666", + title: "MakerLab Tools — mirror", + }) + ), + connect: vi.fn(async (): Promise => ({ ok: true, title: "MakerLab Tools — mirror" })), + ...over, + }; +} + +const tokenField = () => screen.getByLabelText("Integration token"); +const pageField = () => screen.getByLabelText("Page URL"); + +async function fill(user: ReturnType) { + await user.type(tokenField(), TOKEN); + await user.type(pageField(), PAGE_URL); +} + +describe("MirrorConnect", () => { + it("asks for the token in a password field the browser will not autofill", () => { + render(); + expect(tokenField()).toHaveAttribute("type", "password"); + expect(tokenField()).toHaveAttribute("autocomplete", "off"); + expect(tokenField()).toHaveAccessibleDescription(/encrypted before it is stored/); + }); + + it("keeps both buttons disabled until both fields are filled", async () => { + const user = userEvent.setup(); + render(); + + expect(screen.getByRole("button", { name: "Test connection" })).toBeDisabled(); + expect(screen.getByRole("button", { name: "Connect" })).toBeDisabled(); + await user.type(tokenField(), TOKEN); + expect(screen.getByRole("button", { name: "Connect" })).toBeDisabled(); + await user.type(pageField(), PAGE_URL); + expect(screen.getByRole("button", { name: "Connect" })).toBeEnabled(); + }); + + it("tests the connection and shows the page's title, keeping the token for Connect", async () => { + const user = userEvent.setup(); + const bundle = actions(); + render(); + await fill(user); + + await user.click(screen.getByRole("button", { name: "Test connection" })); + + expect(bundle.testConnection).toHaveBeenCalledWith({ token: TOKEN, pageUrl: PAGE_URL }); + expect(await screen.findByText("Found the page “MakerLab Tools — mirror”.")).toBeInTheDocument(); + expect(tokenField()).toHaveValue(TOKEN); + expect(bundle.connect).not.toHaveBeenCalled(); + }); + + it("connects, then clears the token field", async () => { + const user = userEvent.setup(); + const bundle = actions(); + render(); + await fill(user); + + await user.click(screen.getByRole("button", { name: "Connect" })); + + expect(bundle.connect).toHaveBeenCalledWith({ token: TOKEN, pageUrl: PAGE_URL }); + expect(await screen.findByText("Connected to “MakerLab Tools — mirror”.")).toBeInTheDocument(); + expect(tokenField()).toHaveValue(""); + expect(document.body.textContent).not.toContain(TOKEN); + }); + + it("shows a refusal and keeps what was typed so it can be fixed", async () => { + const user = userEvent.setup(); + const bundle = actions({ + connect: vi.fn(async (): Promise => ({ ok: false, error: "page_not_found" })), + }); + render(); + await fill(user); + + await user.click(screen.getByRole("button", { name: "Connect" })); + + expect(await screen.findByText(/Notion could not find that page/)).toBeInTheDocument(); + expect(screen.queryByText(/Connected/)).not.toBeInTheDocument(); + expect(tokenField()).toHaveValue(TOKEN); + }); + + it("renders a gate refusal from the shared admin errors", async () => { + const user = userEvent.setup(); + const bundle = actions({ + testConnection: vi.fn(async (): Promise => ({ ok: false, error: "rate_limited" })), + }); + render(); + await fill(user); + + await user.click(screen.getByRole("button", { name: "Test connection" })); + expect(await screen.findByText(/Too many changes at once/)).toBeInTheDocument(); + }); + + it("forgets a found title once the fields change", async () => { + const user = userEvent.setup(); + render(); + await fill(user); + await user.click(screen.getByRole("button", { name: "Test connection" })); + await screen.findByText(/Found the page/); + + await user.type(pageField(), "x"); + expect(screen.queryByText(/Found the page/)).not.toBeInTheDocument(); + }); + + it("starts from the mirror's page when reconnecting", () => { + render(); + expect(pageField()).toHaveValue("0f5e4a3c-1111-2222-3333-444455556666"); + expect(tokenField()).toHaveValue(""); + }); + + it("shows the audit warning on a connect that landed without its event", async () => { + const user = userEvent.setup(); + const bundle = actions({ + connect: vi.fn( + async (): Promise => ({ ok: true, title: "Mirror", warning: "audit_unavailable" }) + ), + }); + render(); + await fill(user); + + await user.click(screen.getByRole("button", { name: "Connect" })); + expect(await screen.findByText(/could not be written to the audit log/)).toBeInTheDocument(); + expect(tokenField()).toHaveValue(""); + }); +}); diff --git a/v5/src/components/admin/MirrorConnect.tsx b/v5/src/components/admin/MirrorConnect.tsx new file mode 100644 index 0000000..be8b683 --- /dev/null +++ b/v5/src/components/admin/MirrorConnect.tsx @@ -0,0 +1,183 @@ +"use client"; + +import "../../styles/admin-mirror.css"; + +import { useId, useState, type FormEvent } from "react"; +import { useTranslations } from "next-intl"; +import type { AdminActionWarning } from "../../lib/admin/action-result"; +import { + mirrorErrorMessageKey, + type MirrorActionError, + type MirrorActions, +} from "../../app/admin/mirror/action-result"; +import { useRefreshNudge } from "./use-refresh-nudge"; + +/** + * **Connect** (spec §3.8 "Connect", §4.14, §8). + * + * A token field and the URL of the page the integration was shared with. + * **Test connection** reads that page and shows its title, storing nothing; + * **Connect** makes the same read and, only if it succeeds, stores the token + * encrypted. Both are server actions the page hands in, so this island never + * imports an endpoint. + * + * **The token lives in this component's state and nowhere else on the page.** + * The field is `type="password"` with autocomplete off, so the browser neither + * shows nor offers to remember it; it is cleared the moment a connect succeeds; + * and no server answer ever carries it back, so nothing here could echo it + * into a message. On a refusal it stays in the box, because the person is about + * to fix the *page* half as often as the token half. + * + * Every sentence follows an awaited result — a found title, a refusal, a lost + * audit event — and never precedes it (Article 4). + */ + +export type MirrorConnectActions = Pick; + +export interface MirrorConnectProps { + actions: MirrorConnectActions; + /** A page to start from — the mirror's current page when reconnecting. */ + initialPageUrl?: string; +} + +type Outcome = + | { kind: "found"; title: string | null } + | { kind: "connected"; title: string | null; warning: AdminActionWarning | null } + | { kind: "error"; error: MirrorActionError }; + +export function MirrorConnect({ actions, initialPageUrl = "" }: MirrorConnectProps) { + const t = useTranslations("admin"); + const nudge = useRefreshNudge(); + const tokenId = useId(); + const pageId = useId(); + const tokenHintId = `${tokenId}-hint`; + const pageHintId = `${pageId}-hint`; + + const [token, setToken] = useState(""); + const [pageUrl, setPageUrl] = useState(initialPageUrl); + const [busy, setBusy] = useState<"test" | "connect" | null>(null); + const [outcome, setOutcome] = useState(null); + + const filled = token.trim().length > 0 && pageUrl.trim().length > 0; + + async function test() { + setBusy("test"); + setOutcome(null); + try { + const result = await actions.testConnection({ token, pageUrl }); + setOutcome(result.ok ? { kind: "found", title: result.title } : { kind: "error", error: result.error }); + } catch { + setOutcome({ kind: "error", error: "failed" }); + } finally { + setBusy(null); + } + } + + async function connect(event?: FormEvent) { + event?.preventDefault(); + if (!filled || busy) return; + setBusy("connect"); + setOutcome(null); + try { + const result = await actions.connect({ token, pageUrl }); + if (result.ok) { + setToken(""); + setOutcome({ kind: "connected", title: result.title, warning: result.warning ?? null }); + // The page moves to its connected state; see `use-refresh-nudge.ts`. + nudge(); + } else { + setOutcome({ kind: "error", error: result.error }); + } + } catch { + setOutcome({ kind: "error", error: "failed" }); + } finally { + setBusy(null); + } + } + + function edited() { + // A title found for what was typed before says nothing about what is typed now. + if (outcome) setOutcome(null); + } + + let line: string | null = null; + let tone = ""; + if (outcome?.kind === "found") { + line = outcome.title ? t("mirror.connect.found", { title: outcome.title }) : t("mirror.connect.foundUntitled"); + tone = " is-ok"; + } else if (outcome?.kind === "connected") { + line = outcome.warning + ? t(`warnings.${outcome.warning}`) + : outcome.title + ? t("mirror.connect.connected", { title: outcome.title }) + : t("mirror.connect.connectedUntitled"); + tone = outcome.warning ? " is-warning" : " is-ok"; + } else if (outcome?.kind === "error") { + line = t(mirrorErrorMessageKey(outcome.error)); + tone = " is-error"; + } + + return ( +
+

{t("mirror.connect.title")}

+
void connect(event)} noValidate> +
+ + { + setToken(event.target.value); + edited(); + }} + /> + + {t("mirror.connect.tokenHint")} + +
+
+ + { + setPageUrl(event.target.value); + edited(); + }} + /> + + {t("mirror.connect.pageHint")} + +
+
+ + +
+

+ {line} +

+
+
+ ); +} diff --git a/v5/src/components/admin/MirrorControls.test.tsx b/v5/src/components/admin/MirrorControls.test.tsx new file mode 100644 index 0000000..bdda2b4 --- /dev/null +++ b/v5/src/components/admin/MirrorControls.test.tsx @@ -0,0 +1,181 @@ +import { render, screen, userEvent } from "../../../test/utils/render"; +import type { MirrorActionResult } from "../../app/admin/mirror/action-result"; +import type { MirrorView } from "../../lib/mirror/types"; +import { MirrorControls, type MirrorControlActions } from "./MirrorControls"; + +/** + * Sync now, Pause / Resume and Disconnect (spec §3.8 "Controls", §8, Article 4). + * + * The promises: Sync now is disabled with its reason — above all the + * once-per-15-minutes one, with the time left — rather than failing on click; + * a refusal from the server is shown, never a success over it; and Disconnect + * asks first, inline. + */ + +function view(over: Partial = {}): MirrorView { + return { + id: "6c8f3a3e-6f0b-4d62-9d39-2a8b2f7b1e01", + connected: true, + parentPageId: "0f5e4a3c-1111-2222-3333-44445555aaaa", + parentPageTitle: "MakerLab Tools — mirror", + mapping: { tools: "1a2b3c4d-0000-4000-8000-000000000001" }, + paused: false, + running: false, + syncPending: false, + pushScheduled: false, + lastSyncedAt: null, + lastRunAt: null, + lastStatus: null, + lastError: null, + syncAvailableAt: null, + ...over, + }; +} + +function actions(over: Partial = {}): MirrorControlActions { + return { + syncNow: vi.fn(async (): Promise => ({ ok: true })), + setPaused: vi.fn(async (): Promise => ({ ok: true })), + disconnect: vi.fn(async (): Promise => ({ ok: true })), + ...over, + }; +} + +const syncButton = () => screen.getByRole("button", { name: "Sync now" }); + +describe("MirrorControls", () => { + it("starts a sync and says so only once the server agreed", async () => { + const user = userEvent.setup(); + const bundle = actions(); + render(); + + expect(syncButton()).toBeEnabled(); + await user.click(syncButton()); + + expect(bundle.syncNow).toHaveBeenCalledTimes(1); + expect(await screen.findByText("Sync started.")).toBeInTheDocument(); + }); + + it("disables Sync now inside the 15-minute window, with the reason and the time left", () => { + // Half a minute short of ten: the clock is read in 15-second steps, so this is 9.5–9.75 minutes away. + const inTenMinutes = new Date(Date.now() + 10 * 60_000 - 30_000).toISOString(); + render(); + + expect(syncButton()).toBeDisabled(); + const reason = screen.getByText(/Sync now runs once every 15 minutes\./); + expect(reason).toHaveTextContent("Available again in 10 minutes."); + expect(syncButton()).toHaveAccessibleDescription(reason.textContent ?? ""); + }); + + it("enables Sync now once the window has passed", () => { + const aMinuteAgo = new Date(Date.now() - 60_000).toISOString(); + render(); + expect(syncButton()).toBeEnabled(); + }); + + it("shows the server's refusal and how long is left, and no success", async () => { + const user = userEvent.setup(); + const bundle = actions({ + syncNow: vi.fn(async (): Promise => ({ + ok: false, + error: "sync_too_soon", + retryAfterSeconds: 7 * 60, + })), + }); + render(); + + await user.click(syncButton()); + const status = await screen.findByText(/Sync now runs once every 15 minutes\. Available again in 7 minutes\./); + expect(status).toHaveAttribute("role", "status"); + expect(screen.queryByText("Sync started.")).not.toBeInTheDocument(); + }); + + it("renders a gate refusal from the shared admin errors", async () => { + const user = userEvent.setup(); + const bundle = actions({ + syncNow: vi.fn(async (): Promise => ({ ok: false, error: "not_permitted" })), + }); + render(); + + await user.click(syncButton()); + expect(await screen.findByText("Your account does not hold the permission this needs.")).toBeInTheDocument(); + }); + + it("treats a rejected call as failed", async () => { + const user = userEvent.setup(); + const bundle = actions({ syncNow: vi.fn(async () => Promise.reject(new Error("network"))) }); + render(); + + await user.click(syncButton()); + expect(await screen.findByText("That did not save. Nothing was changed.")).toBeInTheDocument(); + }); + + it("disables Sync now on a paused mirror, and on one with no databases, saying why", () => { + const { unmount } = render(); + expect(syncButton()).toBeDisabled(); + expect(screen.getByText("Resume the mirror to sync it.")).toBeInTheDocument(); + unmount(); + + render(); + expect(syncButton()).toBeDisabled(); + expect(screen.getByText("Create or map the databases first.")).toBeInTheDocument(); + }); + + it("pauses and resumes", async () => { + const user = userEvent.setup(); + const bundle = actions(); + const { unmount } = render(); + + await user.click(screen.getByRole("button", { name: "Pause" })); + expect(bundle.setPaused).toHaveBeenCalledWith({ paused: true }); + expect(await screen.findByText("Paused.")).toBeInTheDocument(); + unmount(); + + render(); + await user.click(screen.getByRole("button", { name: "Resume" })); + expect(bundle.setPaused).toHaveBeenLastCalledWith({ paused: false }); + }); + + it("asks before disconnecting, and Cancel changes nothing", async () => { + const user = userEvent.setup(); + const bundle = actions(); + render(); + + await user.click(screen.getByRole("button", { name: "Disconnect" })); + expect(screen.getByText(/Forget the token\?/)).toBeInTheDocument(); + expect(bundle.disconnect).not.toHaveBeenCalled(); + + await user.click(screen.getByRole("button", { name: "Cancel" })); + expect(screen.queryByText(/Forget the token\?/)).not.toBeInTheDocument(); + expect(bundle.disconnect).not.toHaveBeenCalled(); + + await user.click(screen.getByRole("button", { name: "Disconnect" })); + await user.click(screen.getByRole("button", { name: "Yes, disconnect" })); + expect(bundle.disconnect).toHaveBeenCalledTimes(1); + }); + + it("keeps the confirmation open and says why when Disconnect is refused", async () => { + const user = userEvent.setup(); + const bundle = actions({ + disconnect: vi.fn(async (): Promise => ({ ok: false, error: "rate_limited" })), + }); + render(); + + await user.click(screen.getByRole("button", { name: "Disconnect" })); + await user.click(screen.getByRole("button", { name: "Yes, disconnect" })); + expect(await screen.findByText(/Too many changes at once/)).toBeInTheDocument(); + expect(screen.getByRole("button", { name: "Yes, disconnect" })).toBeInTheDocument(); + }); + + it("shows the audit warning on a disconnect that landed without its event", async () => { + const user = userEvent.setup(); + const bundle = actions({ + disconnect: vi.fn(async (): Promise => ({ ok: true, warning: "audit_unavailable" })), + }); + render(); + + await user.click(screen.getByRole("button", { name: "Disconnect" })); + await user.click(screen.getByRole("button", { name: "Yes, disconnect" })); + expect(await screen.findByText(/could not be written to the audit log/)).toBeInTheDocument(); + }); +}); diff --git a/v5/src/components/admin/MirrorControls.tsx b/v5/src/components/admin/MirrorControls.tsx new file mode 100644 index 0000000..d339bb2 --- /dev/null +++ b/v5/src/components/admin/MirrorControls.tsx @@ -0,0 +1,218 @@ +"use client"; + +import "../../styles/admin-mirror.css"; + +import { useId, useState, useSyncExternalStore } from "react"; +import { useTranslations } from "next-intl"; +import type { AdminActionWarning } from "../../lib/admin/action-result"; +import type { MirrorView } from "../../lib/mirror/types"; +import { + mirrorErrorMessageKey, + type MirrorActionError, + type MirrorActionResult, + type MirrorActions, +} from "../../app/admin/mirror/action-result"; +import { useRefreshNudge } from "./use-refresh-nudge"; + +/** + * **Sync now**, **Pause** / **Resume** and **Disconnect** (spec §3.8 + * "Controls", §8). + * + * **Sync now says why it cannot.** One push per mirror per 15 minutes (§8), so + * after a sync the button is disabled with the reason and the time left beside + * it — computed from `syncAvailableAt`, which Postgres worked out against the + * same clock the claim will refuse on. A paused mirror, a mirror with no + * databases and a push already running each disable it with their own reason. + * The server still checks every one of these; the button is presentation. + * + * **Nothing here claims a success it did not get** (Article 4). Every message + * follows the awaited result: a refusal shows `admin.mirror.errors.` (or + * the gate's `admin.errors.`), a lost audit event shows its warning, and + * the page's re-render after `revalidatePath` is what moves the controls on + * (with `useRefreshNudge`, so that re-render is committed). No + * `useTransition`, so the confirmation is never held hostage by that + * re-render (the `/admin/users` lesson). + * + * **Disconnect asks first, inline** — never a modal — and says what it keeps. + */ + +export type MirrorControlActions = Pick; + +export interface MirrorControlsProps { + view: MirrorView; + actions: MirrorControlActions; +} + +type Busy = "sync" | "pause" | "disconnect" | null; +type Done = "syncStarted" | "pausedDone" | "resumedDone" | null; + +/** How often the "available again in" line re-reads the clock. */ +const CLOCK_TICK_MS = 15_000; + +function subscribeClock(onTick: () => void): () => void { + const timer = setInterval(onTick, CLOCK_TICK_MS); + return () => clearInterval(timer); +} + +/** Quantised, so the snapshot is stable between ticks as React requires. */ +function readClock(): number { + return Math.floor(Date.now() / CLOCK_TICK_MS) * CLOCK_TICK_MS; +} + +/** No clock on the server: the first render says "not yet" without a number, then hydrates one in. */ +function readServerClock(): number | null { + return null; +} + +export function MirrorControls({ view, actions }: MirrorControlsProps) { + const t = useTranslations("admin"); + const now = useSyncExternalStore(subscribeClock, readClock, readServerClock); + const reasonId = useId(); + const nudge = useRefreshNudge(); + + const [busy, setBusy] = useState(null); + const [done, setDone] = useState(null); + const [error, setError] = useState(null); + const [retryAfterSeconds, setRetryAfterSeconds] = useState(null); + const [warning, setWarning] = useState(null); + const [confirming, setConfirming] = useState(false); + + const mapped = Object.keys(view.mapping).length > 0; + const availableAt = view.syncAvailableAt ? Date.parse(view.syncAvailableAt) : null; + // Before hydration there is no clock, so the server's own answer stands: a + // `syncAvailableAt` at all means "not yet". + const windowClosed = availableAt !== null && (now === null || availableAt > now); + const minutesLeft = + availableAt !== null && now !== null ? Math.max(1, Math.ceil((availableAt - now) / 60_000)) : null; + + let syncReason: string | null = null; + if (view.paused) syncReason = t("mirror.controls.syncNeedsResume"); + else if (!mapped) syncReason = t("mirror.controls.syncNeedsMapping"); + else if (windowClosed) { + syncReason = + minutesLeft !== null + ? `${t("mirror.controls.syncLimit")} ${t("mirror.controls.syncAvailableIn", { minutes: minutesLeft })}` + : t("mirror.controls.syncLimit"); + } else if (view.running) syncReason = t("mirror.controls.syncRunning"); + + async function perform(which: Exclude, success: Done, call: () => Promise) { + setBusy(which); + setDone(null); + setError(null); + setRetryAfterSeconds(null); + setWarning(null); + try { + const result = await call(); + if (result.ok) { + setDone(success); + setWarning(result.warning ?? null); + // The page re-renders with the change; see `use-refresh-nudge.ts`. + nudge(); + return true; + } + setError(result.error); + setRetryAfterSeconds(result.retryAfterSeconds ?? null); + return false; + } catch { + setError("failed"); + return false; + } finally { + setBusy(null); + } + } + + async function disconnect() { + const ok = await perform("disconnect", null, () => actions.disconnect()); + if (ok) setConfirming(false); + } + + return ( +
+
+ + + {view.connected && !confirming ? ( + + ) : null} +
+ + {syncReason ? ( +

+ {syncReason} +

+ ) : null} + + {confirming ? ( +
+

{t("mirror.controls.disconnectConfirm")}

+ + +
+ ) : null} + +

+ {error + ? [ + t(mirrorErrorMessageKey(error)), + retryAfterSeconds !== null + ? t("mirror.controls.syncAvailableIn", { minutes: Math.max(1, Math.ceil(retryAfterSeconds / 60)) }) + : null, + ] + .filter(Boolean) + .join(" ") + : warning + ? t(`warnings.${warning}`) + : done + ? t(`mirror.controls.${done}`) + : null} +

+
+ ); +} diff --git a/v5/src/components/admin/MirrorMapping.test.tsx b/v5/src/components/admin/MirrorMapping.test.tsx new file mode 100644 index 0000000..af52b5c --- /dev/null +++ b/v5/src/components/admin/MirrorMapping.test.tsx @@ -0,0 +1,152 @@ +import { render, screen, userEvent, within } from "../../../test/utils/render"; +import type { MirrorActionResult, MirrorCreateResult } from "../../app/admin/mirror/action-result"; +import { MirrorMapping, type MirrorMappingActions } from "./MirrorMapping"; + +/** + * The mapping panel (spec §3.8 "Mapping", §5.8, Article 4). + * + * The seven tables in push order, each with its database or "Not set"; Create + * databases saying what it made — including when it stopped part-way, which + * still made some; and a pasted mapping whose problems land under the table + * they concern, naming the properties. + */ + +const TOOLS_DB = "1a2b3c4d-0000-4000-8000-000000000001"; + +function actions(over: Partial = {}): MirrorMappingActions { + return { + createDatabases: vi.fn( + async (): Promise => ({ + ok: true, + created: ["categories", "locations", "units", "resources", "maintenance", "projects"], + kept: ["tools"], + }) + ), + saveMapping: vi.fn(async (): Promise => ({ ok: true })), + ...over, + }; +} + +function rows() { + const table = screen.getByRole("table", { name: "Notion databases, one per table" }); + return within(table).getAllByRole("row").slice(1); +} + +describe("MirrorMapping", () => { + it("lists the seven tables in push order, each with its database or Not set", () => { + render(); + + const listed = rows(); + expect(listed.map((row) => within(row).getByRole("rowheader").textContent)).toEqual([ + "Categories", + "Locations", + "Tools", + "Units", + "Resources", + "Maintenance", + "Projects", + ]); + expect(within(listed[2]).getByText(TOOLS_DB)).toBeInTheDocument(); + expect(within(listed[0]).getByText("Not set")).toBeInTheDocument(); + expect(screen.getAllByText("Not set")).toHaveLength(6); + }); + + it("creates the databases and says how many it made", async () => { + const user = userEvent.setup(); + const bundle = actions(); + render(); + + await user.click(screen.getByRole("button", { name: "Create databases" })); + + expect(bundle.createDatabases).toHaveBeenCalledTimes(1); + expect(await screen.findByText("Created 6 databases.")).toBeInTheDocument(); + }); + + it("says when every database already existed", async () => { + const user = userEvent.setup(); + const bundle = actions({ + createDatabases: vi.fn(async (): Promise => ({ ok: true, created: [], kept: ["tools"] })), + }); + render(); + + await user.click(screen.getByRole("button", { name: "Create databases" })); + expect(await screen.findByText(/Every database already exists/)).toBeInTheDocument(); + }); + + it("names what it made before a create stopped part-way, and where it stopped", async () => { + const user = userEvent.setup(); + const bundle = actions({ + createDatabases: vi.fn( + async (): Promise => ({ + ok: false, + error: "notion_unavailable", + created: ["categories", "locations"], + entity: "tools", + }) + ), + }); + render(); + + await user.click(screen.getByRole("button", { name: "Create databases" })); + + const line = await screen.findByText(/Notion did not answer/); + expect(line).toHaveTextContent("It stopped at Tools."); + expect(line).toHaveTextContent("Created 2 databases before stopping: Categories and Locations."); + expect(line).toHaveClass("is-warning"); + }); + + it("saves pasted ids, sending only the tables that were filled", async () => { + const user = userEvent.setup(); + const bundle = actions(); + render(); + + await user.click(screen.getByText("Use databases you already have")); + await user.type(screen.getByLabelText("Tools database"), ` https://www.notion.so/acme/${TOOLS_DB.replace(/-/g, "")} `); + await user.click(screen.getByRole("button", { name: "Save mapping" })); + + expect(bundle.saveMapping).toHaveBeenCalledWith({ + tools: `https://www.notion.so/acme/${TOOLS_DB.replace(/-/g, "")}`, + }); + expect(await screen.findByText("Saved. Every pasted database matches.")).toBeInTheDocument(); + }); + + it("puts each problem under the table it concerns, naming the properties", async () => { + const user = userEvent.setup(); + const bundle = actions({ + saveMapping: vi.fn( + async (): Promise => ({ + ok: false, + error: "schema_mismatch", + problems: [ + { entity: "tools", code: "schema_mismatch", missing: ["Published", "Slug"], wrongType: ["Category"] }, + { entity: "units", code: "database_not_found" }, + ], + }) + ), + }); + render(); + + await user.click(screen.getByText("Use databases you already have")); + await user.type(screen.getByLabelText("Tools database"), TOOLS_DB); + await user.type(screen.getByLabelText("Units database"), "0f5e4a3c111122223333444455556666"); + await user.click(screen.getByRole("button", { name: "Save mapping" })); + + expect(await screen.findByText(/do not match what the mirror writes/)).toBeInTheDocument(); + const tools = screen.getByLabelText("Tools database"); + expect(tools).toHaveAttribute("aria-invalid", "true"); + expect(tools).toHaveAccessibleDescription(/Missing: Published and Slug/); + expect(tools).toHaveAccessibleDescription(/Wrong type: Category/); + expect(screen.getByLabelText("Units database")).toHaveAccessibleDescription(/Not found/); + expect(screen.getByLabelText("Categories database")).not.toHaveAttribute("aria-invalid"); + expect(screen.queryByText(/^Saved/)).not.toBeInTheDocument(); + }); + + it("shows the mapping but offers no changes without a working token", () => { + render(); + + expect(screen.getByText(TOOLS_DB)).toBeInTheDocument(); + expect(screen.queryByRole("button", { name: "Create databases" })).not.toBeInTheDocument(); + expect(screen.queryByText("Use databases you already have")).not.toBeInTheDocument(); + expect(screen.getByText(/Connect again to change the databases/)).toBeInTheDocument(); + }); +}); diff --git a/v5/src/components/admin/MirrorMapping.tsx b/v5/src/components/admin/MirrorMapping.tsx new file mode 100644 index 0000000..2a1ae58 --- /dev/null +++ b/v5/src/components/admin/MirrorMapping.tsx @@ -0,0 +1,242 @@ +"use client"; + +import "../../styles/admin-mirror.css"; + +import { useId, useState, type FormEvent } from "react"; +import { useFormatter, useTranslations } from "next-intl"; +import { MIRROR_ENTITY, type MirrorEntity } from "../../lib/db/schema/vocabulary"; +import type { MappingProblem, MirrorMapping as MirrorMappingValue } from "../../lib/mirror/types"; +import { + mirrorErrorMessageKey, + type MirrorActionFailure, + type MirrorActions, + type MirrorMappingInput, +} from "../../app/admin/mirror/action-result"; +import { useRefreshNudge } from "./use-refresh-nudge"; + +/** + * The mirror's databases, one per table (spec §3.8 "Mapping", §5.8). + * + * The fixed list of seven, in the order the push writes them (`MIRROR_ENTITY` + * is declared in dependency order), each with its Notion database id or "Not + * set". **Create databases** makes every missing one under the connected page + * — which is also how a database somebody deleted by hand comes back (§5.8): + * only the missing ones are made. An admin who already has databases pastes + * their URLs instead, and **Save mapping** checks each against the schema the + * push writes, saving nothing unless every one matches; a mismatch names the + * properties that are missing or of the wrong type, under the table it is + * about. + * + * **A partial create is said, not hidden** (Article 4). Creation that stops + * part-way has still made databases and written them into the mapping; the + * refusal names them, and the page's re-render shows their ids. + * + * `editable` is false when there is no working token: the mapping is kept and + * shown, so reconnecting creates no duplicates, but nothing can change it + * until the token does. + */ + +export type MirrorMappingActions = Pick; + +export interface MirrorMappingProps { + mapping: MirrorMappingValue; + editable: boolean; + actions: MirrorMappingActions; +} + +type CreateOutcome = + | { kind: "created"; count: number } + | { kind: "failed"; failure: MirrorActionFailure }; + +type SaveOutcome = { kind: "saved" } | { kind: "failed"; failure: MirrorActionFailure }; + +const EMPTY_PASTE: Record = Object.fromEntries( + MIRROR_ENTITY.map((entity) => [entity, ""]) +) as Record; + +export function MirrorMapping({ mapping, editable, actions }: MirrorMappingProps) { + const t = useTranslations("admin"); + const format = useFormatter(); + const formId = useId(); + const nudge = useRefreshNudge(); + + const [busy, setBusy] = useState<"create" | "save" | null>(null); + const [created, setCreated] = useState(null); + const [saved, setSaved] = useState(null); + const [paste, setPaste] = useState>(EMPTY_PASTE); + + const entityName = (entity: MirrorEntity) => t(`mirror.entities.${entity}`); + const names = (entities: MirrorEntity[]) => format.list(entities.map(entityName)); + + async function create() { + setBusy("create"); + setCreated(null); + setSaved(null); + try { + const result = await actions.createDatabases(); + setCreated(result.ok ? { kind: "created", count: result.created.length } : { kind: "failed", failure: result }); + // New ids are in the mapping, even after a partial create; see `use-refresh-nudge.ts`. + if (result.ok || (result.created?.length ?? 0) > 0) nudge(); + } catch { + setCreated({ kind: "failed", failure: { ok: false, error: "failed" } }); + } finally { + setBusy(null); + } + } + + async function save(event: FormEvent) { + event.preventDefault(); + setBusy("save"); + setSaved(null); + setCreated(null); + const input: MirrorMappingInput = {}; + for (const entity of MIRROR_ENTITY) { + const value = paste[entity].trim(); + if (value) input[entity] = value; + } + try { + const result = await actions.saveMapping(input); + if (result.ok) { + setSaved({ kind: "saved" }); + setPaste(EMPTY_PASTE); + nudge(); + } else { + setSaved({ kind: "failed", failure: result }); + } + } catch { + setSaved({ kind: "failed", failure: { ok: false, error: "failed" } }); + } finally { + setBusy(null); + } + } + + const problems = new Map(); + if (saved?.kind === "failed") for (const problem of saved.failure.problems ?? []) problems.set(problem.entity, problem); + + function problemLines(problem: MappingProblem): string[] { + const lines = [t(`mirror.mapping.problems.${problem.code}`)]; + if (problem.missing?.length) lines.push(t("mirror.mapping.missing", { names: format.list(problem.missing) })); + if (problem.wrongType?.length) lines.push(t("mirror.mapping.wrongType", { names: format.list(problem.wrongType) })); + return lines; + } + + let createLine: string | null = null; + let createTone = ""; + if (created?.kind === "created") { + createLine = + created.count > 0 ? t("mirror.mapping.created", { count: created.count }) : t("mirror.mapping.allExisted"); + createTone = " is-ok"; + } else if (created?.kind === "failed") { + const { failure } = created; + const parts = [t(mirrorErrorMessageKey(failure.error))]; + if (failure.entity) parts.push(t("mirror.mapping.stoppedAt", { entity: entityName(failure.entity) })); + if (failure.created?.length) { + parts.push( + t("mirror.mapping.createdBeforeStopping", { count: failure.created.length, names: names(failure.created) }) + ); + } + createLine = parts.join(" "); + // Something was made, so this is not a clean refusal: warn rather than err. + createTone = failure.created?.length ? " is-warning" : " is-error"; + } + + return ( +
+

{t("mirror.mapping.title")}

+ + + + + + + + + + {MIRROR_ENTITY.map((entity) => ( + + + + + ))} + +
{t("mirror.mapping.columnTable")}{t("mirror.mapping.columnDatabase")}
{entityName(entity)} + {mapping[entity] ? ( + {mapping[entity]} + ) : ( + {t("mirror.mapping.notSet")} + )} +
+ + {editable ? ( + <> +

{t("mirror.mapping.createHint")}

+
+ +
+

+ {createLine} +

+ +
+ {t("mirror.mapping.pasteTitle")} +
void save(event)} noValidate> +

{t("mirror.mapping.pasteHint")}

+
+ {MIRROR_ENTITY.map((entity) => { + const id = `${formId}-${entity}`; + const problem = problems.get(entity); + return ( +
+ + setPaste((current) => ({ ...current, [entity]: event.target.value }))} + /> + {problem ? ( +
    + {problemLines(problem).map((line) => ( +
  • {line}
    • + ))} +
+ ) : null} +
+ ); + })} +
+
+ +
+

+ {saved?.kind === "saved" + ? t("mirror.mapping.saved") + : saved?.kind === "failed" + ? t(mirrorErrorMessageKey(saved.failure.error)) + : null} +

+
+
+ + ) : ( +

{t("mirror.mapping.readOnly")}

+ )} +
+ ); +} diff --git a/v5/src/components/admin/MirrorStatus.test.tsx b/v5/src/components/admin/MirrorStatus.test.tsx new file mode 100644 index 0000000..fffafa7 --- /dev/null +++ b/v5/src/components/admin/MirrorStatus.test.tsx @@ -0,0 +1,197 @@ +const router = vi.hoisted(() => ({ refresh: vi.fn() })); + +vi.mock("next/navigation", () => ({ + useRouter: () => router, +})); + +import { act, render, screen, within } from "../../../test/utils/render"; +import { MIRROR_POLL_INTERVAL_MS } from "../../lib/mirror/limits"; +import type { MirrorView } from "../../lib/mirror/types"; +import { MirrorStatus } from "./MirrorStatus"; + +/** + * The mirror's status panel (spec §3.8 "Status", §5.8, §10: "`MirrorStatus`: + * shows last synced, paused, and the last error"). + * + * What it must never do is let a reader infer the wrong thing: a mirror that + * never pushed says "Never", not a blank; a paused one says so in words; a + * push that ran out of time says the rest waits for the next push, and which + * tables it concerned; and while a push is running the page keeps asking for + * a fresh render until it is not. + */ + +const TZ = "America/New_York"; + +function view(over: Partial = {}): MirrorView { + return { + id: "6c8f3a3e-6f0b-4d62-9d39-2a8b2f7b1e01", + connected: true, + parentPageId: "0f5e4a3c-1111-2222-3333-44445555aaaa", + parentPageTitle: "MakerLab Tools — mirror", + mapping: { tools: "1a2b3c4d-0000-4000-8000-000000000001" }, + paused: false, + running: false, + syncPending: false, + pushScheduled: false, + lastSyncedAt: null, + lastRunAt: null, + lastStatus: null, + lastError: null, + syncAvailableAt: null, + ...over, + }; +} + +/** The value beside a `
` label. */ +function fact(label: string): HTMLElement { + const term = screen.getByText(label, { selector: "dt" }); + return term.nextElementSibling as HTMLElement; +} + +beforeEach(() => { + router.refresh.mockClear(); +}); + +afterEach(() => { + vi.useRealTimers(); +}); + +describe("MirrorStatus", () => { + it("shows when it last synced, in the lab's time zone, and the result", () => { + render( + + ); + + const synced = within(fact("Last synced")).getByText(/2026/); + expect(synced.tagName).toBe("TIME"); + expect(synced).toHaveAttribute("datetime", "2026-09-23T14:05:00.000Z"); + // 14:05 UTC is 10:05 in New York in September — formatted, not raw ISO. + expect(synced).toHaveTextContent(/Sep 23, 2026/); + expect(synced).toHaveTextContent(/10:05/); + // The time is a safety watermark, so the panel says what it guarantees. + expect(fact("Last synced")).toHaveTextContent("Every change made before this time is in Notion."); + expect(within(fact("Last run")).getByText(/10:10/)).toBeInTheDocument(); + expect(fact("Last result")).toHaveTextContent("OK"); + expect(screen.getByText("Connected to “MakerLab Tools — mirror”.")).toBeInTheDocument(); + }); + + it("never says Connected once the token has been forgotten", () => { + render(); + + expect(screen.getByText("Disconnected. It last pushed to “MakerLab Tools — mirror”.")).toBeInTheDocument(); + expect(screen.queryByText(/^Connected to/)).not.toBeInTheDocument(); + }); + + it("says a mirror that never pushed has never synced", () => { + render(); + + expect(fact("Last synced")).toHaveTextContent("Never"); + expect(fact("Last run")).toHaveTextContent("Never"); + expect(fact("Last result")).toHaveTextContent("No push yet"); + expect(screen.getByText(/Nothing has been pushed yet/)).toBeInTheDocument(); + expect(screen.queryByText("Last error")).not.toBeInTheDocument(); + }); + + it("says a paused mirror is paused, and shows the error that paused it", () => { + render( + + ); + + expect(screen.getByText("Paused. Nothing is pushed until you resume.")).toBeInTheDocument(); + expect(fact("Last result")).toHaveTextContent("Failed"); + expect(screen.getByText("Last error")).toBeInTheDocument(); + expect(screen.getByText(/Notion refused the token, so the mirror paused itself/)).toBeInTheDocument(); + expect(screen.getByText("Notion 401 unauthorized: API token is invalid.")).toBeInTheDocument(); + }); + + it("shows a partial push that ran out of time, with the tables and rows it concerned", () => { + render( + + ); + + expect(fact("Last result")).toHaveTextContent("Partial"); + expect(screen.getByText(/ran out of time before it finished/)).toBeInTheDocument(); + expect(screen.getByText("Tables: Units and Maintenance")).toBeInTheDocument(); + expect(screen.getByText("3 rows failed.")).toBeInTheDocument(); + // `last_synced_at` did not advance on a partial push (§3.8 step 5), and the + // panel shows the older time rather than pretending the push finished. + expect(within(fact("Last synced")).getByText(/Sep 22, 2026/)).toBeInTheDocument(); + }); + + it("reads an error code it does not know as unknown rather than a raw key", () => { + render( + + ); + expect(screen.getByText(/for a reason the app does not recognise/)).toBeInTheDocument(); + expect(screen.queryByText(/Tables:/)).not.toBeInTheDocument(); + }); + + it("says a push is running, and polls until it is not", () => { + vi.useFakeTimers(); + const { rerender } = render(); + + expect(screen.getByText("Pushing to Notion now…")).toBeInTheDocument(); + act(() => { + vi.advanceTimersByTime(MIRROR_POLL_INTERVAL_MS * 2); + }); + expect(router.refresh).toHaveBeenCalledTimes(2); + + rerender(); + router.refresh.mockClear(); + act(() => { + vi.advanceTimersByTime(MIRROR_POLL_INTERVAL_MS * 3); + }); + expect(router.refresh).not.toHaveBeenCalled(); + }); + + it("polls while a Sync now waits to start, and says so", () => { + vi.useFakeTimers(); + render(); + + expect(screen.getByText(/Sync requested/)).toBeInTheDocument(); + act(() => { + vi.advanceTimersByTime(MIRROR_POLL_INTERVAL_MS); + }); + expect(router.refresh).toHaveBeenCalledTimes(1); + }); + + it("does not poll for a push merely scheduled after recent changes, but says it is coming", () => { + vi.useFakeTimers(); + render(); + + expect(screen.getByText(/Recent changes will be pushed/)).toBeInTheDocument(); + act(() => { + vi.advanceTimersByTime(MIRROR_POLL_INTERVAL_MS * 3); + }); + expect(router.refresh).not.toHaveBeenCalled(); + }); +}); diff --git a/v5/src/components/admin/MirrorStatus.tsx b/v5/src/components/admin/MirrorStatus.tsx new file mode 100644 index 0000000..bc2a51d --- /dev/null +++ b/v5/src/components/admin/MirrorStatus.tsx @@ -0,0 +1,149 @@ +"use client"; + +import "../../styles/admin-mirror.css"; + +import { useRouter } from "next/navigation"; +import { useEffect } from "react"; +import { useFormatter, useTranslations } from "next-intl"; +import { MIRROR_ENTITY, MIRROR_STATUS, type MirrorEntity } from "../../lib/db/schema/vocabulary"; +import { MIRROR_POLL_INTERVAL_MS } from "../../lib/mirror/limits"; +import { MIRROR_ERROR_CODES, type MirrorErrorCode, type MirrorView } from "../../lib/mirror/types"; + +/** + * What the mirror last did, and what it is doing now (spec §3.8 "Status", + * §5.8, §6). + * + * Last synced, last run, last result and the last error — the error in the + * page's words (`admin.mirror.lastError.`), with the tables it concerned + * and Notion's own scrubbed sentence under it, because whoever fixes a mirror + * needs to know *which* database somebody deleted. Paused, running, a Sync now + * waiting to start and a push scheduled after recent edits are each said + * outright rather than left to be inferred from timestamps. + * + * **It polls while a push is running or requested, and only then** — as + * `IntakeList` does. The push happens in a workflow the page cannot subscribe + * to, so it asks for a fresh render every `MIRROR_POLL_INTERVAL_MS`; the + * interval is cleared the moment neither is true, and on unmount. + * + * **Times are formatted by next-intl in the lab's time zone**, passed in by + * the page, never with `Date.toLocaleString()` and never in whatever zone the + * server or the browser happens to be in — so the server's render and the + * browser's agree, and "last synced" means the same hour to everyone reading it. + */ + +export interface MirrorStatusProps { + view: MirrorView; + /** An IANA zone (`LAB_TIMEZONE`), so server and browser render the same time. */ + timeZone: string; +} + +export function MirrorStatus({ view, timeZone }: MirrorStatusProps) { + const t = useTranslations("admin.mirror"); + const format = useFormatter(); + const router = useRouter(); + + const polling = view.running || view.syncPending; + useEffect(() => { + if (!polling) return; + const timer = setInterval(() => router.refresh(), MIRROR_POLL_INTERVAL_MS); + return () => clearInterval(timer); + }, [polling, router]); + + function when(iso: string | null) { + if (!iso) return {t("status.never")}; + const date = new Date(iso); + if (Number.isNaN(date.getTime())) return {t("status.never")}; + return ( + + ); + } + + const status = view.lastStatus && MIRROR_STATUS.includes(view.lastStatus) ? view.lastStatus : null; + const error = view.lastError; + const errorCode: MirrorErrorCode = + error && (MIRROR_ERROR_CODES as readonly string[]).includes(error.code) ? error.code : "unknown"; + const errorEntities = (error?.entities ?? []).filter((entity): entity is MirrorEntity => + (MIRROR_ENTITY as readonly string[]).includes(entity) + ); + + return ( +
+

{t("status.title")}

+ +

+ {/* A disconnected mirror keeps its page and mapping, but saying + "Connected" over a forgotten token would be the lie Article 4 is about. */} + {view.connected + ? view.parentPageTitle + ? t("connectedTo", { title: view.parentPageTitle }) + : t("connectedToUntitled") + : view.parentPageTitle + ? t("disconnectedFrom", { title: view.parentPageTitle }) + : t("disconnectedFromUntitled")} +

+ +
    + {view.paused ?
  • {t("status.paused")}
    • : null} + {view.running ?
  • {t("status.running")}
    • : null} + {!view.running && view.syncPending ? ( +
  • {t("status.syncPending")}
    • + ) : null} + {view.pushScheduled && !view.paused ? ( +
  • {t("status.scheduled")}
    • + ) : null} +
+ +
+
+
{t("status.lastSynced")}
+
+ {when(view.lastSyncedAt)} + {/* The stored time is the push's safety watermark — a few minutes + before the push started — so say what it guarantees rather + than let it read as a clock that is wrong by five minutes. */} + {view.lastSyncedAt ? {t("status.lastSyncedHint")} : null} +
+
+
+
{t("status.lastRun")}
+
{when(view.lastRunAt)}
+
+
+
{t("status.lastResult")}
+
+ {status ? ( + {t(`status.result.${status}`)} + ) : ( + {t("status.noResult")} + )} +
+
+
+ + {!view.lastSyncedAt && !view.lastRunAt && !view.running && !view.syncPending ? ( +

{t("status.neverSynced")}

+ ) : null} + + {error ? ( +
+ {t("status.lastError")} +

{t(`lastError.${errorCode}`)}

+ {errorEntities.length > 0 ? ( +

+ {t("status.errorEntities", { + names: format.list(errorEntities.map((entity) => t(`entities.${entity}`))), + })} +

+ ) : null} + {error.failed > 0 ?

{t("status.errorFailed", { count: error.failed })}

: null} + {error.detail ? ( +

+ {t("status.errorDetail")}{" "} + {error.detail} +

+ ) : null} +
+ ) : null} +
+ ); +} diff --git a/v5/src/components/admin/mirror-messages.test.ts b/v5/src/components/admin/mirror-messages.test.ts new file mode 100644 index 0000000..1ec125b --- /dev/null +++ b/v5/src/components/admin/mirror-messages.test.ts @@ -0,0 +1,70 @@ +import en from "../../../messages/en.json"; +import { MIRROR_ENTITY, MIRROR_STATUS } from "../../lib/db/schema/vocabulary"; +import { MIRROR_ERROR_CODES, MIRROR_SETUP_ERRORS, type MappingProblem } from "../../lib/mirror/types"; +import { mirrorErrorMessageKey } from "../../app/admin/mirror/action-result"; + +/** + * Every code the mirror can answer has a sentence (Article 6, spec §6). + * + * The page renders codes, never English from the server: a setup refusal from + * `admin.mirror.errors.`, a stored push error from + * `admin.mirror.lastError.`. A code added to either vocabulary without a + * key would render as the raw key path on the one page whose job is to explain + * what went wrong — so this fails first. + */ + +type Tree = { [key: string]: string | Tree }; + +function lookup(path: string): unknown { + return path.split(".").reduce((node, part) => (node as Tree | undefined)?.[part], en as unknown as Tree); +} + +function expectMessage(path: string) { + const value = lookup(path); + expect(typeof value, `messages/en.json is missing "${path}"`).toBe("string"); + expect((value as string).trim(), `"${path}" is empty`).not.toBe(""); +} + +describe("the mirror's messages", () => { + it.each(MIRROR_SETUP_ERRORS)("has admin.mirror.errors.%s", (code) => { + expectMessage(`admin.mirror.errors.${code}`); + expect(mirrorErrorMessageKey(code)).toBe(`mirror.errors.${code}`); + }); + + it.each(MIRROR_ERROR_CODES)("has admin.mirror.lastError.%s", (code) => { + expectMessage(`admin.mirror.lastError.${code}`); + }); + + it("has a name for every mirrored table and every push result", () => { + for (const entity of MIRROR_ENTITY) expectMessage(`admin.mirror.entities.${entity}`); + for (const status of MIRROR_STATUS) expectMessage(`admin.mirror.status.result.${status}`); + }); + + it("has a sentence for every mapping problem", () => { + const codes: MappingProblem["code"][] = ["invalid_database_id", "database_not_found", "schema_mismatch"]; + for (const code of codes) expectMessage(`admin.mirror.mapping.problems.${code}`); + }); + + it("renders the gate's codes from the errors every admin surface shares", () => { + for (const code of ["not_signed_in", "not_permitted", "rate_limited", "failed", "invalid_field"] as const) { + expect(mirrorErrorMessageKey(code)).toBe(`errors.${code}`); + expectMessage(`admin.errors.${code}`); + } + }); + + it("lists the mirror on /admin with a title and a lede that take no argument", () => { + // `/admin/page.tsx` renders every lede without arguments (Article 6). + expectMessage("admin.mirrorTitle"); + expectMessage("admin.mirrorLede"); + expect(lookup("admin.mirrorLede")).not.toMatch(/[{}]/); + }); + + it("puts no placeholder in a refusal, which is rendered without arguments", () => { + for (const code of MIRROR_SETUP_ERRORS) expect(lookup(`admin.mirror.errors.${code}`)).not.toMatch(/[{}]/); + for (const code of MIRROR_ERROR_CODES) expect(lookup(`admin.mirror.lastError.${code}`)).not.toMatch(/[{}]/); + }); + + it("no longer promises the mirror for a later phase", () => { + expect(lookup("admin.indexMoreComing")).toBeUndefined(); + }); +}); diff --git a/v5/src/components/admin/use-refresh-nudge.test.tsx b/v5/src/components/admin/use-refresh-nudge.test.tsx new file mode 100644 index 0000000..ed39ce0 --- /dev/null +++ b/v5/src/components/admin/use-refresh-nudge.test.tsx @@ -0,0 +1,58 @@ +import { act, renderHook } from "@testing-library/react"; +import { REFRESH_NUDGE_DURATION_MS, REFRESH_NUDGE_INTERVAL_MS, useRefreshNudge } from "./use-refresh-nudge"; + +/** + * `useRefreshNudge` re-renders its island for a few seconds after an action, + * so a refreshed page that finished rendering is committed (see the module's + * docstring). What matters: it renders while nudging, it stops on its own, and + * it stops on unmount — an island the new page no longer contains must not + * keep a timer alive. + */ + +let renders = 0; + +function mount() { + renders = 0; + return renderHook(() => { + renders += 1; + return useRefreshNudge(); + }); +} + +beforeEach(() => { + vi.useFakeTimers(); +}); + +afterEach(() => { + vi.useRealTimers(); +}); + +it("does nothing until nudged", () => { + mount(); + act(() => vi.advanceTimersByTime(REFRESH_NUDGE_DURATION_MS)); + expect(renders).toBe(1); +}); + +it("re-renders on every interval while nudging, then stops by itself", () => { + const { result } = mount(); + + act(() => result.current()); + // One act per tick: inside a single act React would batch the three bumps. + for (let tick = 0; tick < 3; tick++) act(() => vi.advanceTimersByTime(REFRESH_NUDGE_INTERVAL_MS)); + expect(renders).toBe(4); + + act(() => vi.advanceTimersByTime(REFRESH_NUDGE_DURATION_MS * 2)); + const settled = renders; + expect(settled).toBeLessThanOrEqual(2 + REFRESH_NUDGE_DURATION_MS / REFRESH_NUDGE_INTERVAL_MS); + expect(vi.getTimerCount()).toBe(0); + + act(() => vi.advanceTimersByTime(REFRESH_NUDGE_DURATION_MS)); + expect(renders).toBe(settled); +}); + +it("stops when the island unmounts", () => { + const { result, unmount } = mount(); + act(() => result.current()); + unmount(); + expect(vi.getTimerCount()).toBe(0); +}); diff --git a/v5/src/components/admin/use-refresh-nudge.ts b/v5/src/components/admin/use-refresh-nudge.ts new file mode 100644 index 0000000..e0f5bd6 --- /dev/null +++ b/v5/src/components/admin/use-refresh-nudge.ts @@ -0,0 +1,57 @@ +"use client"; + +import { useCallback, useEffect, useReducer, useRef } from "react"; + +/** How often a nudge re-renders the island while a refreshed page may be waiting. */ +export const REFRESH_NUDGE_INTERVAL_MS = 200; +/** How long the nudging lasts — far longer than the refreshed tree takes to arrive. */ +export const REFRESH_NUDGE_DURATION_MS = 4_000; + +/** + * Make sure the page a server action refreshed is actually shown. + * + * **What goes wrong without it.** The mirror's actions end in + * `revalidatePath("/admin/mirror")`, so the action's response carries the + * re-rendered page and Next applies it in a transition. In this app's + * production build (Next 16.1, React 19.2, `cacheComponents`) that transition + * finished rendering and then **was not committed until something else updated + * the page** — a keystroke, a click, a timer. Found by E2E scenario 8: after + * Connect the form stayed on screen indefinitely, yet typing one character + * swapped in the connected page within 25 ms; an in-place refresh (Pause → + * Resume) sat 11–14 s until `MirrorControls`' 15-second clock ticked. Nothing + * was pending on the network, `requestAnimationFrame` fired, fonts were + * loaded. A later `router.refresh()` queued behind the stalled one, so calling + * it does not help. + * + * **What this does.** The island calls `nudge()` once its action has + * answered. For the next few seconds it re-renders itself every + * {@link REFRESH_NUDGE_INTERVAL_MS}; each render is an ordinary update, which + * makes React pick up the finished transition and commit it. The island is + * small, the renders change nothing it shows, and the interval stops on + * unmount — which is exactly what happens when the committed page no longer + * contains it (Connect, Disconnect). + * + * If a future Next or React release commits the transition on its own, this + * becomes a harmless handful of no-op renders; delete it then, and E2E + * scenario 8 will say whether that was right. + */ +export function useRefreshNudge(): () => void { + const [, bump] = useReducer((count: number) => count + 1, 0); + const timer = useRef | null>(null); + + const stop = useCallback(() => { + if (timer.current !== null) clearInterval(timer.current); + timer.current = null; + }, []); + + useEffect(() => stop, [stop]); + + return useCallback(() => { + stop(); + const until = Date.now() + REFRESH_NUDGE_DURATION_MS; + timer.current = setInterval(() => { + bump(); + if (Date.now() >= until) stop(); + }, REFRESH_NUDGE_INTERVAL_MS); + }, [stop]); +} diff --git a/v5/src/components/catalog-types.ts b/v5/src/components/catalog-types.ts index f0f1ced..452b921 100644 --- a/v5/src/components/catalog-types.ts +++ b/v5/src/components/catalog-types.ts @@ -36,6 +36,11 @@ export interface MakerLabTool { href: string; kind?: string; description?: string; + /** + * The manufacturer's link, when `href` is the archived copy of it in Blob + * (the manual archive). Kept so a caller can still name the original. + */ + sourceHref?: string; }>; units: MakerLabUnit[]; } diff --git a/v5/src/lib/blob-local.test.ts b/v5/src/lib/blob-local.test.ts new file mode 100644 index 0000000..355bfd0 --- /dev/null +++ b/v5/src/lib/blob-local.test.ts @@ -0,0 +1,164 @@ +// @vitest-environment node +import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { getBlobStore, isBlobConfigured } from "./blob"; +import { + createLocalBlobBackend, + localBlobUrl, + resolveLocalPath, +} from "./blob-local"; +import { createBlobUploader } from "./import/blob-uploader"; + +/** + * The `.blob-data/` store against a temporary folder. `process.cwd()` is + * pointed at the folder so the default root is exercised too; nothing touches + * the working tree. + */ + +let dir: string; + +beforeEach(async () => { + dir = await mkdtemp(join(tmpdir(), "blob-local-")); + vi.spyOn(process, "cwd").mockReturnValue(dir); + vi.stubEnv("BLOB_READ_WRITE_TOKEN", ""); + vi.stubEnv("VERCEL", ""); + vi.stubEnv("NODE_ENV", "development"); + vi.stubEnv("BLOB_LOCAL_DISABLE", ""); + vi.stubEnv("AUTH_BASE_URL", "http://localhost:3001/"); +}); + +afterEach(async () => { + await rm(dir, { recursive: true, force: true }); +}); + +function photo(name = "broken bed.png", type = "image/png") { + return new File([new Uint8Array([1, 2, 3])], name, { type }); +} + +describe("local mode through the BlobStore seam", () => { + it("counts as configured, and is not configured once disabled", () => { + expect(isBlobConfigured()).toBe(true); + vi.stubEnv("BLOB_LOCAL_DISABLE", "1"); + expect(isBlobConfigured()).toBe(false); + }); + + it("put writes a private file at exactly its pathname and overwrites on re-run", async () => { + const store = getBlobStore(); + expect(await store.put("backups/2026-07-29.json", "{}", "application/json")).toEqual({ + pathname: "backups/2026-07-29.json", + }); + await store.put("backups/2026-07-29.json", '{"a":1}', "application/json"); + + expect(await readFile(join(dir, ".blob-data/backups/2026-07-29.json"), "utf8")).toBe('{"a":1}'); + const read = await createLocalBlobBackend().read("backups/2026-07-29.json"); + expect(read?.meta).toMatchObject({ access: "private", contentType: "application/json", size: 7 }); + }); + + it("putUpload stores at a random pathname under the prefix, with a dev-blob URL", async () => { + const stored = await getBlobStore().putUpload("uploads/project/", photo(), "public"); + + expect(stored.pathname).toMatch(/^uploads\/project\/broken-bed-[A-Za-z0-9]{8,}\.png$/); + expect(stored.url).toBe(`http://localhost:3001/api/dev-blob/${stored.pathname}`); + const read = await createLocalBlobBackend().read(stored.pathname); + expect(read?.meta).toMatchObject({ access: "public", contentType: "image/png", size: 3 }); + expect([...read!.body]).toEqual([1, 2, 3]); + }); + + it("two uploads of the same name never collide", async () => { + const store = getBlobStore(); + const a = await store.putUpload("uploads/chat/", photo("IMG_0001.jpg", "image/jpeg"), "private"); + const b = await store.putUpload("uploads/chat/", photo("IMG_0001.jpg", "image/jpeg"), "private"); + expect(a.pathname).not.toBe(b.pathname); + }); + + it("strips path separators out of an untrusted filename", async () => { + const stored = await getBlobStore().putUpload("uploads/project/", photo("../../etc/passwd.png"), "public"); + expect(stored.pathname).toMatch(/^uploads\/project\/etc-passwd-[A-Za-z0-9]+\.png$/); + }); + + it("copyToPublic makes a public copy at a new random pathname and leaves the source", async () => { + const store = getBlobStore(); + const source = await store.putUpload("uploads/chat/", photo("plate.jpg", "image/jpeg"), "private"); + const copy = await store.copyToPublic(source.pathname, "uploads/tool/"); + + expect(copy.pathname.startsWith("uploads/tool/plate-")).toBe(true); + expect(copy.pathname).not.toBe(`uploads/tool/${source.pathname.split("/").pop()}`); + const disk = createLocalBlobBackend(); + expect((await disk.read(copy.pathname))?.meta).toMatchObject({ access: "public", contentType: "image/jpeg" }); + expect((await disk.read(source.pathname))?.meta.access).toBe("private"); + }); + + it("copyToPublic rejects for a missing source rather than reporting a URL", async () => { + await expect(getBlobStore().copyToPublic("uploads/chat/nope.jpg", "uploads/tool/")).rejects.toThrow(); + }); + + it("list returns every blob under a prefix, and del removes them", async () => { + const store = getBlobStore(); + await store.put("backups/a.json", "{}", "application/json"); + await store.put("backups/b.json", "{}", "application/json"); + await store.putUpload("uploads/chat/", photo(), "private"); + + const backups = await store.list("backups/"); + expect(backups.map((b) => b.pathname)).toEqual(["backups/a.json", "backups/b.json"]); + expect(backups[0].uploadedAt).toMatch(/^\d{4}-\d{2}-\d{2}T/); + expect(await store.list("uploads/")).toHaveLength(1); + + await store.del(["backups/a.json"]); + expect((await store.list("backups/")).map((b) => b.pathname)).toEqual(["backups/b.json"]); + await store.del([]); + }); + + it("list is empty before anything has been written", async () => { + expect(await getBlobStore().list("backups/")).toEqual([]); + }); +}); + +describe("the backend's own rules", () => { + it("refuses to overwrite unless allowed", async () => { + const disk = createLocalBlobBackend(); + await disk.put("a/b.txt", "1", { access: "private" }); + await expect(disk.put("a/b.txt", "2", { access: "private" })).rejects.toThrow("already exists"); + }); + + it("refuses traversal, absolute paths and the metadata folder", async () => { + const disk = createLocalBlobBackend(); + for (const bad of ["../escape.txt", "a/../../escape.txt", "/etc/passwd", ".meta/x.json", "a//b", "a\\b", ""]) { + await expect(disk.put(bad, "x", { access: "public" })).rejects.toThrow("Invalid blob pathname"); + expect(() => resolveLocalPath(dir, bad)).toThrow(); + } + expect(await createLocalBlobBackend().list("")).toEqual([]); + }); + + it("encodes each segment of a URL", () => { + expect(localBlobUrl("uploads/a b#c.png")).toBe("http://localhost:3001/api/dev-blob/uploads/a%20b%23c.png"); + }); + + it("falls back to localhost:3000 without AUTH_BASE_URL", () => { + vi.stubEnv("AUTH_BASE_URL", ""); + expect(localBlobUrl("x.pdf")).toBe("http://localhost:3000/api/dev-blob/x.pdf"); + }); +}); + +describe("createBlobUploader (the step-code path)", () => { + it("is the local store in local mode, with a random suffix", async () => { + const uploader = createBlobUploader(); + expect(uploader).not.toBeNull(); + const stored = await uploader!.put("manuals/t1/r1.pdf", new TextEncoder().encode("%PDF-1.7"), { + access: "public", + contentType: "application/pdf", + }); + expect(stored.pathname).toMatch(/^manuals\/t1\/r1-[A-Za-z0-9]+\.pdf$/); + expect(stored.url).toBe(`http://localhost:3001/api/dev-blob/${stored.pathname}`); + expect((await createLocalBlobBackend().read(stored.pathname))?.meta).toMatchObject({ + access: "public", + contentType: "application/pdf", + size: 8, + }); + }); + + it("is null with no store at all", () => { + vi.stubEnv("BLOB_LOCAL_DISABLE", "1"); + expect(createBlobUploader()).toBeNull(); + }); +}); diff --git a/v5/src/lib/blob-local.ts b/v5/src/lib/blob-local.ts new file mode 100644 index 0000000..52f461d --- /dev/null +++ b/v5/src/lib/blob-local.ts @@ -0,0 +1,221 @@ +import { randomBytes } from "node:crypto"; +import { mkdir, readdir, readFile, rm, stat, writeFile } from "node:fs/promises"; +import { dirname, join, relative, resolve, sep } from "node:path"; + +/** + * A folder that behaves like Vercel Blob, for a laptop with no + * `BLOB_READ_WRITE_TOKEN` (`blobMode() === "local"`, see `blob-mode.ts`). + * + * Bytes live at `.blob-data/`; what the real store would remember + * about each one — access, content type, size, upload time — lives beside them + * in `.blob-data/.meta/.json`. The folder is git-ignored. + * + * It keeps the store's semantics that callers rely on: a random suffix when + * asked for one (so `putUpload` and `copyToPublic` return a pathname the caller + * did not choose), a refusal to overwrite unless allowed, and public vs private + * access — a public file is served by `GET /api/dev-blob/`, a private + * one never is. + * + * Not `server-only` (the manual archiver runs under plain Node), and never used + * unless `blobMode()` says "local", which it cannot on Vercel or in a + * production build. + */ + +export type LocalBlobAccess = "public" | "private"; + +export interface LocalBlobMeta { + access: LocalBlobAccess; + contentType: string; + size: number; + uploadedAt: string; +} + +export interface LocalPutOptions { + access: LocalBlobAccess; + contentType?: string; + addRandomSuffix?: boolean; + allowOverwrite?: boolean; +} + +export interface LocalBlobBackend { + put( + pathname: string, + body: string | Uint8Array | Blob, + options: LocalPutOptions + ): Promise<{ pathname: string; url: string }>; + copy( + from: string, + to: string, + options: { access: LocalBlobAccess; addRandomSuffix?: boolean } + ): Promise<{ pathname: string; url: string }>; + list(prefix: string): Promise<{ pathname: string; uploadedAt: string }[]>; + del(pathnames: string[]): Promise; + /** The bytes and metadata, or null when there is no such blob. */ + read(pathname: string): Promise<{ body: Uint8Array; meta: LocalBlobMeta } | null>; +} + +/** Where the folder is: `.blob-data/` in the working directory (v5/ under `next dev`). */ +export function localBlobRoot(): string { + return join(process.cwd(), ".blob-data"); +} + +const META_DIR = ".meta"; + +/** + * The origin written into a public file's URL. `AUTH_BASE_URL` is what local + * development already sets to the dev server's own address (it is the OAuth + * return origin); without it, Next's default port. + */ +export function localBlobOrigin(): string { + const explicit = (process.env.AUTH_BASE_URL || "").trim(); + return (explicit || "http://localhost:3000").replace(/\/$/, ""); +} + +export function localBlobUrl(pathname: string): string { + const encoded = pathname.split("/").map(encodeURIComponent).join("/"); + return `${localBlobOrigin()}/api/dev-blob/${encoded}`; +} + +/** + * Validate a pathname and map it into the folder. Refuses anything that could + * leave the folder or reach the metadata: empty, `.` / `..` or dot-leading + * segments, backslashes, NUL, absolute paths. Throws on refusal. + */ +export function resolveLocalPath(root: string, pathname: string): string { + if (typeof pathname !== "string" || pathname.length === 0 || pathname.length > 1024) { + throw new Error("Invalid blob pathname"); + } + if (pathname.includes("\\") || pathname.includes("\0") || pathname.startsWith("/")) { + throw new Error("Invalid blob pathname"); + } + const segments = pathname.split("/"); + if (segments.some((s) => s.length === 0 || s.startsWith("."))) { + throw new Error("Invalid blob pathname"); + } + const base = resolve(root); + const full = resolve(base, ...segments); + if (!full.startsWith(base + sep)) throw new Error("Invalid blob pathname"); + return full; +} + +function metaPath(root: string, pathname: string): string { + // `pathname` has already been through resolveLocalPath. + return join(resolve(root), META_DIR, `${pathname}.json`); +} + +/** `photo.jpg` → `photo-.jpg`, as Vercel Blob's `addRandomSuffix` does. */ +function withRandomSuffix(pathname: string): string { + const slash = pathname.lastIndexOf("/"); + const dir = pathname.slice(0, slash + 1); + const name = pathname.slice(slash + 1); + const dot = name.lastIndexOf("."); + const suffix = randomBytes(16).toString("base64url").replace(/[-_]/g, "").slice(0, 20); + if (dot <= 0) return `${dir}${name}-${suffix}`; + return `${dir}${name.slice(0, dot)}-${suffix}${name.slice(dot)}`; +} + +async function toBytes(body: string | Uint8Array | Blob): Promise { + if (typeof body === "string") return new TextEncoder().encode(body); + if (body instanceof Uint8Array) return body; + return new Uint8Array(await body.arrayBuffer()); +} + +async function exists(path: string): Promise { + try { + await stat(path); + return true; + } catch { + return false; + } +} + +async function readMeta(root: string, pathname: string): Promise { + try { + return JSON.parse(await readFile(metaPath(root, pathname), "utf8")) as LocalBlobMeta; + } catch { + return null; + } +} + +export function createLocalBlobBackend(root: string = localBlobRoot()): LocalBlobBackend { + async function write( + requested: string, + bytes: Uint8Array, + options: LocalPutOptions + ): Promise<{ pathname: string; url: string }> { + const pathname = options.addRandomSuffix ? withRandomSuffix(requested) : requested; + const file = resolveLocalPath(root, pathname); + if (!options.allowOverwrite && (await exists(file))) { + throw new Error(`This blob already exists: ${pathname}`); + } + const meta: LocalBlobMeta = { + access: options.access, + contentType: options.contentType || "application/octet-stream", + size: bytes.byteLength, + uploadedAt: new Date().toISOString(), + }; + await mkdir(dirname(file), { recursive: true }); + await writeFile(file, bytes); + const metaFile = metaPath(root, pathname); + await mkdir(dirname(metaFile), { recursive: true }); + await writeFile(metaFile, JSON.stringify(meta)); + return { pathname, url: localBlobUrl(pathname) }; + } + + async function read(pathname: string) { + const file = resolveLocalPath(root, pathname); + const meta = await readMeta(root, pathname); + if (!meta) return null; + try { + return { body: new Uint8Array(await readFile(file)), meta }; + } catch { + return null; + } + } + + return { + async put(pathname, body, options) { + return write(pathname, await toBytes(body), options); + }, + + async copy(from, to, options) { + const source = await read(from); + if (!source) throw new Error(`The requested blob does not exist: ${from}`); + return write(to, source.body, { + access: options.access, + contentType: source.meta.contentType, + addRandomSuffix: options.addRandomSuffix, + }); + }, + + async list(prefix) { + const base = resolve(root); + let entries: string[]; + try { + entries = (await readdir(base, { recursive: true, withFileTypes: true })) + .filter((e) => e.isFile()) + .map((e) => relative(base, join(e.parentPath, e.name)).split(sep).join("/")); + } catch { + return []; + } + const blobs: { pathname: string; uploadedAt: string }[] = []; + for (const pathname of entries.sort()) { + if (pathname.startsWith(`${META_DIR}/`) || !pathname.startsWith(prefix)) continue; + const meta = await readMeta(root, pathname); + if (!meta) continue; + blobs.push({ pathname, uploadedAt: meta.uploadedAt }); + } + return blobs; + }, + + async del(pathnames) { + for (const pathname of pathnames) { + const file = resolveLocalPath(root, pathname); + await rm(file, { force: true }); + await rm(metaPath(root, pathname), { force: true }); + } + }, + + read, + }; +} diff --git a/v5/src/lib/blob-mode.test.ts b/v5/src/lib/blob-mode.test.ts new file mode 100644 index 0000000..2f8dbc7 --- /dev/null +++ b/v5/src/lib/blob-mode.test.ts @@ -0,0 +1,62 @@ +import { blobMode } from "./blob-mode"; + +/** + * The one rule for which Blob store a process uses. vitest.setup.ts sets + * `BLOB_LOCAL_DISABLE=1` for the suite, so every row states it explicitly. + */ +function env(vars: { + token?: string; + vercel?: string; + nodeEnv?: string; + disable?: string; +}) { + vi.stubEnv("BLOB_READ_WRITE_TOKEN", vars.token ?? ""); + vi.stubEnv("VERCEL", vars.vercel ?? ""); + vi.stubEnv("NODE_ENV", vars.nodeEnv ?? "development"); + vi.stubEnv("BLOB_LOCAL_DISABLE", vars.disable ?? ""); +} + +describe("blobMode", () => { + it("is vercel whenever a token is set, wherever it runs", () => { + env({ token: "vercel_blob_rw_test" }); + expect(blobMode()).toBe("vercel"); + env({ token: "vercel_blob_rw_test", vercel: "1", nodeEnv: "production" }); + expect(blobMode()).toBe("vercel"); + }); + + it("is none on Vercel without a token — a deploy never falls back to disk", () => { + env({ vercel: "1" }); + expect(blobMode()).toBe("none"); + }); + + it("is none in a production build without a token", () => { + env({ nodeEnv: "production" }); + expect(blobMode()).toBe("none"); + }); + + it("is local in development without a token", () => { + env({}); + expect(blobMode()).toBe("local"); + env({ nodeEnv: "test" }); + expect(blobMode()).toBe("local"); + }); + + it("is none when BLOB_LOCAL_DISABLE is set", () => { + env({ disable: "1" }); + expect(blobMode()).toBe("none"); + env({ disable: "true" }); + expect(blobMode()).toBe("none"); + }); + + it("treats BLOB_LOCAL_DISABLE=0 / false as not disabled", () => { + env({ disable: "0" }); + expect(blobMode()).toBe("local"); + env({ disable: "false" }); + expect(blobMode()).toBe("local"); + }); + + it("is none by default in the test suite", () => { + vi.stubEnv("BLOB_READ_WRITE_TOKEN", ""); + expect(blobMode()).toBe("none"); + }); +}); diff --git a/v5/src/lib/blob-mode.ts b/v5/src/lib/blob-mode.ts new file mode 100644 index 0000000..0803b2c --- /dev/null +++ b/v5/src/lib/blob-mode.ts @@ -0,0 +1,26 @@ +/** + * Which Blob store this process writes to — the one rule, in one place. + * + * - `"vercel"` — `BLOB_READ_WRITE_TOKEN` is set: the real Vercel Blob store. + * - `"local"` — no token, not on Vercel, not a production build, and not + * switched off with `BLOB_LOCAL_DISABLE=1`: files go to `.blob-data/` in the + * working directory (`blob-local.ts`), so uploads, promotion, archived + * manuals, backups and the orphan sweep all work on a laptop. + * - `"none"` — anything else. A deploy without a linked store keeps failing + * loudly (`blob_not_configured`, 503); it never falls back to a disk that + * would vanish with the function instance. The test suites set + * `BLOB_LOCAL_DISABLE=1` (vitest.setup.ts) so "no token" still means "none" + * there unless a test opts in. + * + * Not `server-only`: the manual archiver and the Notion import run under plain + * Node, where that package throws. Relative, `.ts`-suffixed imports only. + */ +export type BlobMode = "vercel" | "local" | "none"; + +export function blobMode(): BlobMode { + if (process.env.BLOB_READ_WRITE_TOKEN) return "vercel"; + if (process.env.VERCEL || process.env.NODE_ENV === "production") return "none"; + const disabled = (process.env.BLOB_LOCAL_DISABLE ?? "").trim().toLowerCase(); + if (disabled && disabled !== "0" && disabled !== "false") return "none"; + return "local"; +} diff --git a/v5/src/lib/blob.ts b/v5/src/lib/blob.ts index a7ced02..59a070a 100644 --- a/v5/src/lib/blob.ts +++ b/v5/src/lib/blob.ts @@ -1,6 +1,8 @@ import "server-only"; import { copy, del, list, put } from "@vercel/blob"; +import { createLocalBlobBackend } from "./blob-local"; +import { blobMode } from "./blob-mode"; /** * Blob storage — one narrow seam over Vercel Blob (ops hardening design spec @@ -86,12 +88,16 @@ export interface BlobStore { /** * `BLOB_READ_WRITE_TOKEN` is injected by Vercel when a Blob store is linked to - * the project, and is absent locally. Callers check this up front so a - * misconfigured deploy fails with a clear answer instead of an SDK error buried - * in a cron log — the whole point of §3.3 is that a backup never fails quietly. + * the project. Callers check this up front so a misconfigured deploy fails with + * a clear answer instead of an SDK error buried in a cron log — the whole point + * of §3.3 is that a backup never fails quietly. + * + * Without a token, local development still has a store: `.blob-data/` on disk + * (`blob-mode.ts` decides; `blob-local.ts` is the folder). On Vercel or in a + * production build there is no such fallback, and this stays false. */ export function isBlobConfigured(): boolean { - return Boolean(process.env.BLOB_READ_WRITE_TOKEN); + return blobMode() !== "none"; } /** Guards a runaway `list` loop; 30 days of daily backups is ~30 blobs. */ @@ -111,7 +117,54 @@ function safeFilename(name: string): string { return cleaned || "upload"; } +/** + * The store for this process: Vercel Blob with a token, the `.blob-data/` + * folder in local development without one. Callers check + * {@link isBlobConfigured} first; the local store follows the same rules as the + * real one (private backups at their exact path, random upload pathnames, + * copy-to-public at a new pathname). + */ export function getBlobStore(): BlobStore { + return blobMode() === "local" ? localBlobStore() : vercelBlobStore(); +} + +function localBlobStore(): BlobStore { + const disk = createLocalBlobBackend(); + return { + async put(pathname, body, contentType) { + const result = await disk.put(pathname, body, { + access: "private", + contentType, + addRandomSuffix: false, + allowOverwrite: true, + }); + return { pathname: result.pathname }; + }, + putUpload(prefix, file, access) { + return disk.put(`${prefix}${safeFilename(file.name)}`, file, { + access, + contentType: file.type || "application/octet-stream", + addRandomSuffix: true, + }); + }, + copyToPublic(pathname, prefix) { + const basename = pathname.slice(pathname.lastIndexOf("/") + 1); + return disk.copy(pathname, `${prefix}${safeFilename(basename)}`, { + access: "public", + addRandomSuffix: true, + }); + }, + list(prefix) { + return disk.list(prefix); + }, + async del(pathnames) { + if (pathnames.length === 0) return; + await disk.del(pathnames); + }, + }; +} + +function vercelBlobStore(): BlobStore { return { async put(pathname, body, contentType) { const result = await put(pathname, body, { diff --git a/v5/src/lib/capabilities/intake.ts b/v5/src/lib/capabilities/intake.ts index 3c82f92..4b983e2 100644 --- a/v5/src/lib/capabilities/intake.ts +++ b/v5/src/lib/capabilities/intake.ts @@ -8,6 +8,7 @@ import { promoteAttachmentsToPublic } from "../files/promote"; import { IDENTIFY_MAX_ITEMS, IDENTIFY_MAX_MODEL_NAME_SEARCHES, RESEARCH_MAX_ITEMS_PER_REQUEST } from "../intake/limits"; import type { DuplicateOf, IntakeTablePayload, IntakeTableWarning } from "../intake/types"; import { toPendingToolView } from "../intake/view"; +import { requestManualArchive } from "../manuals/trigger"; import { verifyResourceLinks } from "../research/verify-links"; import { invalidateCatalog } from "../revalidate"; import { INTAKE_PERMISSION } from "./access"; @@ -411,6 +412,10 @@ const createToolTool: CapabilityTool = { warnings.push("The draft was saved, but the catalogue cache could not be refreshed — it may take a while to appear."); } + // Copy each manual PDF into Blob before the manufacturer moves it. Never + // throws; a run that could not start is the nightly backfill's. + await requestManualArchive(outcome.resourceIds); + return { success: true, tool_id: outcome.toolId, diff --git a/v5/src/lib/cron/backup-policy.test.ts b/v5/src/lib/cron/backup-policy.test.ts index c687137..146d53c 100644 --- a/v5/src/lib/cron/backup-policy.test.ts +++ b/v5/src/lib/cron/backup-policy.test.ts @@ -1,6 +1,6 @@ // @vitest-environment node import { getTableName } from "drizzle-orm"; -import { account, session, tools, user, verification } from "../db/schema/index"; +import { account, notionMirrors, session, tools, user, verification } from "../db/schema/index"; import { EXCLUDED_TABLES, isExcludedFromBackup, redactRows } from "./backup-policy"; /** @@ -72,6 +72,26 @@ describe("redactRows", () => { expect(accountRow.accessToken).toBe("ya29.live-access-token"); }); + it("blanks a mirror's Notion token and keeps what a restore needs", () => { + const mirrorRow = { + id: "m-1", + ownerUserId: "user-1", + tokenCiphertext: Buffer.from([1, 2, 3, 4]), + parentPageId: "page-1", + mapping: { tools: "db-1" }, + lastStatus: "ok", + }; + + const [row] = redactRows(getTableName(notionMirrors), [mirrorRow]) as Record[]; + + // Decryptable by anyone who also holds AUTH_SECRET, and a Buffer besides. + expect(row.tokenCiphertext).toBeNull(); + expect("tokenCiphertext" in row).toBe(true); + expect(row.ownerUserId).toBe("user-1"); + expect(row.mapping).toEqual({ tools: "db-1" }); + expect(isExcludedFromBackup(notionMirrors)).toBe(false); + }); + it("passes a table with no redactions straight through", () => { const rows = [{ id: "t-1", name: "Formlabs Form 3" }]; diff --git a/v5/src/lib/cron/backup-policy.ts b/v5/src/lib/cron/backup-policy.ts index 34a026c..b4a393b 100644 --- a/v5/src/lib/cron/backup-policy.ts +++ b/v5/src/lib/cron/backup-policy.ts @@ -1,6 +1,7 @@ import { getTableName } from "drizzle-orm"; import type { PgTable } from "drizzle-orm/pg-core"; import { account, session, verification } from "../db/schema/auth"; +import { notionMirrors } from "../db/schema/mirror"; /** * What the nightly export deliberately leaves out (data platform design spec @@ -29,6 +30,11 @@ import { account, session, verification } from "../db/schema/auth"; * nothing and keeping them costs a credential in a file. * - **`user` is kept whole, deliberately.** `role` and `banned` are the state * a restore would most need to get right, and the row carries no secret. + * - **`notion_mirrors` is kept, with its token blanked** (Phase 8). The + * ciphertext is decryptable by anyone who also holds `AUTH_SECRET`, and as a + * `bytea` it would serialise as a `Buffer` object besides. A restored mirror + * keeps its mapping and pages and asks its owner for the token again — the + * same thing rotating `AUTH_SECRET` does (spec §8). * * Everything is named through the table objects rather than string literals, so * renaming a table or a column fails the typecheck here instead of quietly @@ -46,6 +52,14 @@ const ACCOUNT_SECRETS = [ "password", ] as const satisfies readonly (keyof typeof account.$inferSelect)[]; +/** + * `notion_mirrors` columns blanked in the export: the owner's Notion token, + * encrypted under a key derived from `AUTH_SECRET`. + */ +const MIRROR_SECRETS = [ + "tokenCiphertext", +] as const satisfies readonly (keyof typeof notionMirrors.$inferSelect)[]; + /** Tables the nightly file does not contain at all. */ export const EXCLUDED_TABLES: ReadonlySet = new Set([ getTableName(session), @@ -55,6 +69,7 @@ export const EXCLUDED_TABLES: ReadonlySet = new Set([ /** Per-table column blanklists, by SQL table name. */ const REDACTED_COLUMNS: Readonly> = { [getTableName(account)]: ACCOUNT_SECRETS, + [getTableName(notionMirrors)]: MIRROR_SECRETS, }; /** True when this table's rows must not be written to a backup file at all. */ diff --git a/v5/src/lib/cron/manual-archive.test.ts b/v5/src/lib/cron/manual-archive.test.ts new file mode 100644 index 0000000..567b36b --- /dev/null +++ b/v5/src/lib/cron/manual-archive.test.ts @@ -0,0 +1,110 @@ +// @vitest-environment node + +/** + * The daily cron's manual stage against PGlite: which manuals are due (Manual, + * a link, no PDF copy of that link), the moving oldest-first window, and the + * counts the route reports. Starting the workflow is mocked at `start.ts`. + */ + +const starter = vi.hoisted(() => ({ startManualArchive: vi.fn() })); +vi.mock("../manuals/start", () => starter); + +import { listManualsDueForArchive, manualSourceKey } from "../data/manual-archives"; +import { createPgliteDb } from "../db/pglite"; +import { attachments, resources, tools } from "../db/schema/index"; +import type { Db } from "../db/types"; +import { MANUALS_PER_NIGHT, runManualArchiveBackfill } from "./manual-archive"; + +let db: Db; +let toolId: string; + +beforeAll(async () => { + db = await createPgliteDb(); +}); + +beforeEach(async () => { + starter.startManualArchive.mockReset().mockResolvedValue(true); + vi.spyOn(console, "info").mockImplementation(() => {}); + vi.spyOn(console, "error").mockImplementation(() => {}); + await db.delete(attachments); + await db.delete(tools); + const [tool] = await db.insert(tools).values({ slug: "p1s", name: "P1S" }).returning({ id: tools.id }); + toolId = tool.id; +}); + +/** `n` Manual resources, created a minute apart, oldest first. */ +async function manuals(n: number, values: Partial = {}): Promise { + const base = Date.UTC(2026, 0, 1); + const rows = await db + .insert(resources) + .values( + Array.from({ length: n }, (_, i) => ({ + toolId, + title: `Manual ${i}`, + type: "Manual", + url: `https://maker.test/m${i}.pdf`, + createdAt: new Date(base + i * 60_000), + ...values, + })) + ) + .returning({ id: resources.id }); + return rows.map((row) => row.id); +} + +async function pdf(resourceId: string, sourceKey: string | null) { + await db.insert(attachments).values({ + ownerType: "resource", + ownerId: resourceId, + blobPathname: `manuals/${resourceId}.pdf`, + access: "public", + publicUrl: `https://blob.test/${resourceId}.pdf`, + contentType: "application/pdf", + sourceKey, + }); +} + +describe("listManualsDueForArchive", () => { + it("picks Manuals with an http link and no PDF of that link, oldest first", async () => { + const [archived, stale, uploaded, due] = await manuals(4); + await pdf(archived, manualSourceKey(archived, "https://maker.test/m0.pdf")); + await pdf(stale, manualSourceKey(stale, "https://maker.test/old.pdf")); + await pdf(uploaded, null); + await db.insert(resources).values([ + { toolId, title: "SOP", type: "SOP", url: "https://maker.test/sop.pdf" }, + { toolId, title: "No link", type: "manual", url: null }, + { toolId, title: "Placeholder", type: "Manual", url: "#" }, + ]); + + expect(await listManualsDueForArchive({ db, limit: 10, day: 0 })).toEqual({ due: 2, ids: [stale, due] }); + }); + + it("moves the window each night and wraps, so every manual is tried", async () => { + const ids = await manuals(5); + + expect((await listManualsDueForArchive({ db, limit: 2, day: 0 })).ids).toEqual([ids[0], ids[1]]); + expect((await listManualsDueForArchive({ db, limit: 2, day: 1 })).ids).toEqual([ids[2], ids[3]]); + expect((await listManualsDueForArchive({ db, limit: 2, day: 2 })).ids).toEqual([ids[4], ids[0]]); + }); +}); + +describe("runManualArchiveBackfill", () => { + it("starts nothing and reports zeros when nothing is due", async () => { + expect(await runManualArchiveBackfill({ db })).toEqual({ due: 0, queued: 0, failed: 0 }); + expect(starter.startManualArchive).not.toHaveBeenCalled(); + }); + + it("hands at most ten manuals to one run and reports the counts", async () => { + const ids = await manuals(12); + + expect(await runManualArchiveBackfill({ db, now: new Date(0) })).toEqual({ due: 12, queued: MANUALS_PER_NIGHT, failed: 0 }); + expect(starter.startManualArchive).toHaveBeenCalledTimes(1); + expect(starter.startManualArchive).toHaveBeenCalledWith(ids.slice(0, MANUALS_PER_NIGHT)); + }); + + it("counts a run that could not be started as failed", async () => { + await manuals(1); + starter.startManualArchive.mockResolvedValue(false); + + expect(await runManualArchiveBackfill({ db })).toEqual({ due: 1, queued: 0, failed: 1 }); + }); +}); diff --git a/v5/src/lib/cron/manual-archive.ts b/v5/src/lib/cron/manual-archive.ts new file mode 100644 index 0000000..0da88e7 --- /dev/null +++ b/v5/src/lib/cron/manual-archive.ts @@ -0,0 +1,51 @@ +import { listManualsDueForArchive } from "../data/manual-archives.ts"; +import type { Db } from "../db/types.ts"; +import { requestManualArchive } from "../manuals/trigger.ts"; + +/** + * The daily cron's manual stage — the backfill and backstop for the manual + * archive (`src/lib/manuals/archive.ts`). + * + * Every night up to {@link MANUALS_PER_NIGHT} Manual resources with a link + * and no PDF copy of it are handed to one `archiveManuals` run: imported + * manuals nobody has archived yet, a resource whose link was edited, an + * approval whose run never started. The window moves each night (see + * `listManualsDueForArchive`), so a manual whose link is an HTML page cannot + * hold the backfill up. + * + * This stage only **starts** the run; the downloads happen in the workflow, + * outside the cron's 60 seconds. A run that could not be started counts as + * `failed`, which the route reports as a failed stage — a backfill that + * silently started nothing is the quiet failure the cron exists to prevent. + */ + +/** Manuals handed to the archive per night. */ +export const MANUALS_PER_NIGHT = 10; + +export interface ManualArchiveStageOptions { + /** A handle to use instead of `getDb()` — tests pass an isolated one. */ + db?: Db; + /** The clock, for the moving window. */ + now?: Date; +} + +export interface ManualArchiveStageResult { + /** Manual resources still without a copy of their link. */ + due: number; + /** Handed to tonight's run. */ + queued: number; + /** 1 when the run could not be started, else 0. */ + failed: number; +} + +export async function runManualArchiveBackfill( + options: ManualArchiveStageOptions = {} +): Promise { + const day = Math.floor((options.now ?? new Date()).getTime() / 86_400_000); + const { due, ids } = await listManualsDueForArchive({ db: options.db, limit: MANUALS_PER_NIGHT, day }); + if (ids.length === 0) return { due, queued: 0, failed: 0 }; + + const started = await requestManualArchive(ids); + console.info(`[cron] manual archive: due=${due} queued=${started ? ids.length : 0} started=${started}`); + return { due, queued: started ? ids.length : 0, failed: started ? 0 : 1 }; +} diff --git a/v5/src/lib/cron/mirror-backstop.test.ts b/v5/src/lib/cron/mirror-backstop.test.ts new file mode 100644 index 0000000..6348faa --- /dev/null +++ b/v5/src/lib/cron/mirror-backstop.test.ts @@ -0,0 +1,78 @@ +// @vitest-environment node + +/** + * The daily cron's mirror stage against PGlite (spec §3.8 trigger 3, §3.9). + * `startMirrorPush` is mocked; which mirrors are due is + * `listMirrorsDueForBackstop`'s own test, so this proves the stage around it: + * every due mirror gets a start, and one that cannot be started is counted + * and does not stop the rest. + */ + +const starter = vi.hoisted(() => ({ startMirrorPush: vi.fn() })); + +vi.mock("../mirror/start", () => ({ startMirrorPush: starter.startMirrorPush })); + +import { sql } from "drizzle-orm"; +import { finishMirrorRun, saveMirrorConnection, setMirrorPaused } from "../data/mirrors"; +import { createPgliteDb } from "../db/pglite"; +import { notionMirrors, user } from "../db/schema/index"; +import type { Db } from "../db/types"; +import { runMirrorBackstop } from "./mirror-backstop"; + +const PAGE = "0f5e4a3c-1111-2222-3333-444455556666"; + +let db: Db; + +beforeAll(async () => { + db = await createPgliteDb(); +}); + +beforeEach(async () => { + starter.startMirrorPush.mockReset().mockResolvedValue({ ok: true, runId: "run" }); + vi.spyOn(console, "info").mockImplementation(() => {}); + await db.delete(notionMirrors); +}); + +async function mirror(): Promise<{ owner: string; id: string }> { + const owner = `u-${crypto.randomUUID()}`; + await db.insert(user).values({ id: owner, name: "Mirror Owner", email: `${owner}@example.test`, role: "admin" }); + const { mirror: created } = await saveMirrorConnection( + { ownerUserId: owner, tokenCiphertext: new Uint8Array([1, 2, 3]), parentPageId: PAGE, parentPageTitle: null }, + { db } + ); + return { owner, id: created.id }; +} + +describe("runMirrorBackstop", () => { + it("starts nothing when there is no mirror", async () => { + expect(await runMirrorBackstop({ db })).toEqual({ due: 0, started: 0, failed: 0 }); + expect(starter.startMirrorPush).not.toHaveBeenCalled(); + }); + + it("starts a push for every due mirror, and none for a paused or up-to-date one", async () => { + const neverSynced = await mirror(); + const paused = await mirror(); + await setMirrorPaused(paused.owner, true, { db }); + const upToDate = await mirror(); + // Synced "in the future", so no source row can be newer than it. + await finishMirrorRun(upToDate.id, { status: "ok", error: null, advanceTo: null, pause: false, generation: 0 }, { db }); + await db.execute(sql`update notion_mirrors set last_synced_at = now() + interval '1 day' where id = ${upToDate.id}`); + + expect(await runMirrorBackstop({ db })).toEqual({ due: 1, started: 1, failed: 0 }); + expect(starter.startMirrorPush).toHaveBeenCalledWith(neverSynced.id); + expect(starter.startMirrorPush).toHaveBeenCalledTimes(1); + }); + + it("counts a start that fails or throws, and carries on with the rest", async () => { + await mirror(); + await mirror(); + await mirror(); + starter.startMirrorPush + .mockResolvedValueOnce({ ok: false }) + .mockRejectedValueOnce(new Error("boom")) + .mockResolvedValueOnce({ ok: true, runId: "run" }); + + expect(await runMirrorBackstop({ db })).toEqual({ due: 3, started: 1, failed: 2 }); + expect(starter.startMirrorPush).toHaveBeenCalledTimes(3); + }); +}); diff --git a/v5/src/lib/cron/mirror-backstop.ts b/v5/src/lib/cron/mirror-backstop.ts new file mode 100644 index 0000000..673c793 --- /dev/null +++ b/v5/src/lib/cron/mirror-backstop.ts @@ -0,0 +1,57 @@ +import { listMirrorsDueForBackstop } from "../data/mirrors.ts"; +import type { Db } from "../db/types.ts"; +import { startMirrorPush } from "../mirror/start.ts"; + +/** + * The daily cron's mirror stage (spec §3.8 trigger 3, §3.9: "pushes any mirror + * whose data is newer than its last sync"). + * + * The backstop for anything the change trigger and Sync now missed: a push + * that failed, one that ran out of rounds, a change made while another push + * held the mirror, a start that never happened. `listMirrorsDueForBackstop` + * picks the mirrors that are active, not running, and either never synced, + * not `ok` last time, or behind a source table that changed since — so a + * quiet night starts nothing. + * + * This stage only **starts** the pushes; each runs in its own `mirrorPush` + * workflow, well outside the cron's 60-second function. One that cannot be + * started counts as `failed` and the rest are still started. The route treats + * a non-zero `failed` as a failed stage, the same as a throw (the database + * unreachable): a backstop that silently started nothing is the quiet failure + * the cron exists to prevent. What each push then did is on its mirror row. + */ + +export interface MirrorBackstopOptions { + /** A handle to use instead of `getDb()` — tests pass an isolated one. */ + db?: Db; +} + +export interface MirrorBackstopResult { + /** Mirrors the backstop found behind. */ + due: number; + /** Pushes started. */ + started: number; + /** Pushes that could not be started. */ + failed: number; +} + +export async function runMirrorBackstop(options: MirrorBackstopOptions = {}): Promise { + const due = await listMirrorsDueForBackstop({ db: options.db }); + let started = 0; + let failed = 0; + + for (const mirrorId of due) { + try { + const result = await startMirrorPush(mirrorId); + if (result.ok) started += 1; + else failed += 1; + } catch { + failed += 1; + } + } + + if (due.length > 0) { + console.info(`[cron] mirror backstop: due=${due.length} started=${started} failed=${failed}`); + } + return { due: due.length, started, failed }; +} diff --git a/v5/src/lib/data/attachments.ts b/v5/src/lib/data/attachments.ts index b35f3b1..1f2815d 100644 --- a/v5/src/lib/data/attachments.ts +++ b/v5/src/lib/data/attachments.ts @@ -277,6 +277,11 @@ export interface NewAttachment { originalFilename: string; /** The signed-in uploader, or null. Anonymous uploads stay allowed (§3.3). */ uploadedBy: string | null; + /** + * An idempotency key, globally unique — set by a copy the app makes on its + * own (an archived manual, `manual::`), never by an upload. + */ + sourceKey?: string | null; } export interface AttachmentReadOptions { @@ -309,6 +314,7 @@ export async function createAttachment( sizeBytes: row.sizeBytes, originalFilename: row.originalFilename, uploadedBy: row.uploadedBy, + sourceKey: row.sourceKey ?? null, }) .returning({ id: attachments.id }); diff --git a/v5/src/lib/data/catalog.test.ts b/v5/src/lib/data/catalog.test.ts index 65b199e..526840e 100644 --- a/v5/src/lib/data/catalog.test.ts +++ b/v5/src/lib/data/catalog.test.ts @@ -11,6 +11,7 @@ import { UNIT_STATUS, } from "../db/schema/index"; import type { Db } from "../db/types"; +import { manualSourceKey } from "./manual-archives"; import { countPublishedTools, deriveTrainingLabel, @@ -751,6 +752,53 @@ describe("resourceLinks", () => { ]); expect(links[0]).toMatchObject({ label: "Resource", kind: "Resource" }); }); + + describe("an archived manual", () => { + const SOURCE = "https://maker.test/form-4-manual.pdf"; + const manual: ResourceRow = { ...resourceRow, title: "Form 4 manual", type: "Manual", url: SOURCE }; + const archive = (url: string, publicUrl = "https://blob.test/manuals/form-4.pdf") => + file({ + ownerType: "resource", + ownerId: "res-id", + publicUrl, + originalFilename: "form-4-manual.pdf", + sourceKey: manualSourceKey("res-id", url), + }); + + it("links the copy once, with the manufacturer's link kept as the source", () => { + expect(resourceLinks([manual], indexAttachments([archive(SOURCE)]))).toEqual([ + { + label: "Form 4 manual", + href: "https://blob.test/manuals/form-4.pdf", + sourceHref: SOURCE, + kind: "Manual", + description: undefined, + }, + ]); + }); + + it("falls back to the source link when there is no copy", () => { + expect(resourceLinks([manual])).toEqual([ + { label: "Form 4 manual", href: SOURCE, kind: "Manual", description: undefined }, + ]); + }); + + it("drops a stale copy of a link the resource no longer carries, rather than listing it", () => { + const links = resourceLinks([manual], indexAttachments([archive("https://maker.test/old.pdf", "https://blob.test/old.pdf")])); + expect(links.map((link) => link.href)).toEqual([SOURCE]); + }); + + it("still lists an uploaded file beside the archived link", () => { + const files = indexAttachments([ + archive(SOURCE), + file({ ownerType: "resource", ownerId: "res-id", publicUrl: "https://blob.test/quick-start.pdf", position: 1 }), + ]); + expect(resourceLinks([manual], files).map((link) => link.href)).toEqual([ + "https://blob.test/manuals/form-4.pdf", + "https://blob.test/quick-start.pdf", + ]); + }); + }); }); describe("localToolImage", () => { diff --git a/v5/src/lib/data/catalog.ts b/v5/src/lib/data/catalog.ts index 51cfa01..d0fc71f 100644 --- a/v5/src/lib/data/catalog.ts +++ b/v5/src/lib/data/catalog.ts @@ -10,6 +10,7 @@ import { } from "../db/schema/index.ts"; import type { Db } from "../db/types.ts"; import { compactNotionId } from "../legacy-id.ts"; +import { isManualArchiveKey, manualSourceKey } from "./manual-archives.ts"; import { isUuid } from "./uuid.ts"; import type { MakerLabTool, MakerLabUnit, ToolStatus } from "../../components/catalog-types.ts"; @@ -81,6 +82,8 @@ export interface AttachmentRow { access: string; publicUrl: string | null; originalFilename: string | null; + /** Set on an archived manual (`manual::`); see `./manual-archives.ts`. */ + sourceKey?: string | null; } /** Attachments grouped by `:`, each list in position order. */ @@ -292,6 +295,7 @@ async function selectAttachments( access: attachments.access, publicUrl: attachments.publicUrl, originalFilename: attachments.originalFilename, + sourceKey: attachments.sourceKey, }) .from(attachments) .where(and(or(ownedByTool, ownedByResource), isNotNull(attachments.publicUrl))) @@ -379,7 +383,16 @@ export function toolImageSrc(tool: Pick, files: AttachmentRow[] return image?.publicUrl || localToolImage(tool.name); } -/** A link per resource url, plus one per file the resource owns. */ +/** + * A link per resource url, plus one per file the resource owns. + * + * **An archived manual replaces its link rather than joining it.** When the + * resource owns a public copy of the PDF its `url` points at (the manual + * archive, `./manual-archives.ts`), the one link goes to the copy — it + * survives the manufacturer moving the file — and the manufacturer's URL rides + * along as `sourceHref`. The copy is never listed as a second file link, and a + * copy of a link the resource no longer carries (stale) is not listed at all. + */ export function resourceLinks( resourceRows: ResourceRow[], files: AttachmentIndex = new Map() @@ -389,19 +402,22 @@ export function resourceLinks( kind: resource.type || "Resource", description: resource.notes || undefined, }; + const owned = attachmentsFor(files, "resource", resource.id); + const archive = resource.url ? archivedCopy(owned, resource.id, resource.url) : undefined; const urlLinks = resource.url ? [ { label: resource.title || resource.type || "Resource", - href: resource.url, + href: archive?.publicUrl || resource.url, + ...(archive ? { sourceHref: resource.url } : {}), ...base, }, ] : []; // A private file has no public URL to link to, and would not be one a // visitor is allowed to open even if it did. - const fileLinks = attachmentsFor(files, "resource", resource.id).flatMap((file) => - file.access === "public" && file.publicUrl + const fileLinks = owned.flatMap((file) => + file.access === "public" && file.publicUrl && !isManualArchiveKey(file.sourceKey) ? [ { label: resource.title || file.originalFilename || resource.type || "Resource", @@ -416,6 +432,12 @@ export function resourceLinks( }); } +/** The resource's public archive of exactly this link, if it has one. */ +function archivedCopy(owned: AttachmentRow[], resourceId: string, url: string): AttachmentRow | undefined { + const key = manualSourceKey(resourceId, url); + return owned.find((file) => file.sourceKey === key && file.access === "public" && Boolean(file.publicUrl)); +} + export function toMakerLabUnit( unit: UnitRow, tool: Pick diff --git a/v5/src/lib/data/maintenance.ts b/v5/src/lib/data/maintenance.ts index 25b35c2..e74682c 100644 --- a/v5/src/lib/data/maintenance.ts +++ b/v5/src/lib/data/maintenance.ts @@ -32,8 +32,10 @@ import { isUuid } from "./uuid.ts"; * trips back to the words Notion showed. * - **The reporter's email is selected by exactly one read.** * `reported_by_email` is set only from a resolved session and must not enter - * a model prompt or the mirror (spec §8, PII), so the history read — whose - * rows a model sees — does not select the column at all. Phase 5's + * a model prompt (spec §8, PII), so the history read — whose rows a model + * sees — does not select the column at all. (The Notion mirror does carry it, + * per the 2026-09-23 amendment, but through its own select in + * `mirror/source.ts`, not through this module.) Phase 5's * {@link listMaintenanceQueue} does, because `/admin/maintenance` is gated on * `maintenance.manage` and answering a confusing ticket means writing back to * the person who filed it. Which read a caller picks is therefore the whole @@ -349,7 +351,9 @@ export interface MaintenanceQueueEntry { * The reporter's address, **only on this projection**. * * {@link listMaintenanceHistoryForUnit} deliberately does not select this - * column, because its rows reach a model prompt and the Notion mirror (§8). + * column, because its rows reach a model prompt (§8). (The Notion mirror + * carries reporter emails since the 2026-09-23 amendment, but it selects + * them in `mirror/source.ts`; neither read here feeds it.) * This read has one caller — a page gated on `maintenance.manage` — and the * first thing an admin does with a confusing ticket is ask the person who * filed it. Showing a name they cannot reach is a queue that sends them back diff --git a/v5/src/lib/data/manual-archives.ts b/v5/src/lib/data/manual-archives.ts new file mode 100644 index 0000000..77d14e0 --- /dev/null +++ b/v5/src/lib/data/manual-archives.ts @@ -0,0 +1,155 @@ +import { and, eq, like, ne, sql } from "drizzle-orm"; +import { getDb } from "../db/client.ts"; +import { rawRows } from "../db/raw.ts"; +import { attachments } from "../db/schema/index.ts"; +import type { Db } from "../db/types.ts"; +import { isUuid } from "./uuid.ts"; + +/** + * Archived manuals — a resource's PDF, copied into Blob so the tool keeps its + * manual after the manufacturer moves or deletes it. + * + * **No table of its own.** The copy is an ordinary `attachments` row owned by + * the resource (`owner_type = 'resource'`), public, `application/pdf`, and + * recognised by its `source_key`: `manual::`. The + * resource keeps its own `url` — the manufacturer's link — so the key says + * *which* link the copy was made from, and a copy of a link the resource no + * longer carries is stale rather than current. The resource id is in the key + * because `source_key` is globally unique and two tools can share a + * manufacturer's PDF. + * + * Relative imports with `.ts` extensions, no `@/` alias and no + * `"server-only"`, like every other module under `src/lib/data/`: the archive + * step runs under plain Node in a workflow. + */ + +/** Every archive key starts with this. */ +export const MANUAL_SOURCE_PREFIX = "manual:"; + +/** + * Longer URLs are not archived: the key would outgrow what a btree unique + * index accepts, and a 2 000-character manual link is not one worth guessing + * about. + */ +export const MAX_ARCHIVABLE_URL_LENGTH = 2000; + +/** The `source_key` of this resource's archived copy of `url`. */ +export function manualSourceKey(resourceId: string, url: string): string { + return `${MANUAL_SOURCE_PREFIX}${resourceId}:${url}`; +} + +/** True for any archive key, current or stale. */ +export function isManualArchiveKey(sourceKey: string | null | undefined): boolean { + return typeof sourceKey === "string" && sourceKey.startsWith(MANUAL_SOURCE_PREFIX); +} + +/** A PDF the resource already holds, as the archiver needs to see it. */ +export interface ResourcePdf { + id: string; + sourceKey: string | null; +} + +/** Every `application/pdf` attachment this resource owns. */ +export async function listResourcePdfs(db: Db, resourceId: string): Promise { + if (!isUuid(resourceId)) return []; + return db + .select({ id: attachments.id, sourceKey: attachments.sourceKey }) + .from(attachments) + .where( + and( + eq(attachments.ownerType, "resource"), + eq(attachments.ownerId, resourceId), + eq(attachments.contentType, "application/pdf") + ) + ); +} + +/** + * Take this resource's archives of *other* links off it, so the daily sweep + * collects them (`releaseAttachments`' rule: never delete bytes from inside a + * write). Called in the transaction that records the new copy, after the + * resource's link was edited. + */ +export async function releaseStaleManualArchives(db: Db, resourceId: string, currentKey: string): Promise { + if (!isUuid(resourceId)) return 0; + const rows = await db + .update(attachments) + .set({ ownerType: null, ownerId: null, position: 0 }) + .where( + and( + eq(attachments.ownerType, "resource"), + eq(attachments.ownerId, resourceId), + like(attachments.sourceKey, `${MANUAL_SOURCE_PREFIX}${resourceId}:%`), + ne(attachments.sourceKey, currentKey) + ) + ) + .returning({ id: attachments.id }); + return rows.length; +} + +export interface ManualsDueOptions { + /** A handle to use instead of {@link getDb} — tests pass an isolated one. */ + db?: Db; + /** How many to return. */ + limit: number; + /** + * Which night this is, as a day count. The window rotates by it — see + * {@link listManualsDueForArchive}. + */ + day: number; +} + +/** + * Manual resources with a link and no PDF for it yet — the daily cron's + * backfill (and backstop for a start that never happened). + * + * Due means: `type` is Manual (any case), `url` is an http(s) link short + * enough to key, and the resource owns no `application/pdf` attachment that + * is either its archive of *this* link or a file somebody uploaded or the + * import copied. A stale archive (of a link since edited) does not count. + * + * **Oldest first, in a window that moves each night.** Nothing records that + * an archive was refused — a manual whose link is an HTML product page stays + * due forever — so "the ten oldest" would be the same ten refusals every + * night and the backfill would never reach the eleventh. Instead the window + * starts at `(day × limit) mod due` in oldest-first order and wraps, so every + * due manual is tried within `ceil(due / limit)` nights whatever fails. + */ +export async function listManualsDueForArchive(options: ManualsDueOptions): Promise<{ due: number; ids: string[] }> { + const db = options.db ?? (await getDb()); + const limit = Math.max(0, Math.floor(options.limit)); + + const where = sql` + lower(r.type) = 'manual' + and r.url ~* '^https?://' + and length(r.url) <= ${MAX_ARCHIVABLE_URL_LENGTH} + and not exists ( + select 1 from attachments a + where a.owner_type = 'resource' + and a.owner_id = r.id + and a.content_type = 'application/pdf' + and (a.source_key is null + or a.source_key not like ${`${MANUAL_SOURCE_PREFIX}%`} + or a.source_key = ${MANUAL_SOURCE_PREFIX} || r.id::text || ':' || r.url) + )`; + + const [{ count }] = await rawRows<{ count: number | string }>( + db, + sql`select count(*)::int as count from resources r where ${where}` + ); + const due = Number(count); + if (due === 0 || limit === 0) return { due, ids: [] }; + + const start = ((Math.floor(options.day) * limit) % due + due) % due; + const page = (offset: number, take: number) => + rawRows<{ id: string }>( + db, + sql`select r.id from resources r where ${where} + order by r.created_at asc, r.id asc + offset ${offset} limit ${take}` + ); + + const first = await page(start, limit); + const rest = first.length < limit && start > 0 ? await page(0, Math.min(limit - first.length, start)) : []; + return { due, ids: [...first, ...rest].map((row) => row.id) }; +} diff --git a/v5/src/lib/data/mirror-pages.test.ts b/v5/src/lib/data/mirror-pages.test.ts new file mode 100644 index 0000000..4f78b80 --- /dev/null +++ b/v5/src/lib/data/mirror-pages.test.ts @@ -0,0 +1,135 @@ +// @vitest-environment node +import { eq, sql } from "drizzle-orm"; +import { createPgliteDb } from "../db/pglite"; +import { rawRows } from "../db/raw"; +import { categories, maintenanceLogs, mirrorPages, notionMirrors, tools, user } from "../db/schema/index"; +import type { Db } from "../db/types"; +import { + MIRROR_SOURCE_TABLE, + deleteMirrorPage, + getMirrorPageIds, + listOrphanedMirrorPages, + upsertMirrorPage, +} from "./mirror-pages"; + +describe("mirror_pages", () => { + let db: Db; + let mirrorId: string; + let otherMirrorId: string; + + beforeAll(async () => { + db = await createPgliteDb(); + const ids: string[] = []; + for (const owner of ["u-pages-1", "u-pages-2"]) { + await db.insert(user).values({ id: owner, name: "Owner", email: `${owner}@cornell.edu`, role: "admin" }); + const [row] = await db + .insert(notionMirrors) + .values({ ownerUserId: owner, parentPageId: "page" }) + .returning({ id: notionMirrors.id }); + ids.push(row.id); + } + [mirrorId, otherMirrorId] = ids; + }); + + it("records a page, finds it, and re-records it in place", async () => { + const entityId = crypto.randomUUID(); + await upsertMirrorPage( + { mirrorId, entity: "tools", entityId, notionPageId: "n-1", sourceUpdatedAt: "2026-09-23 10:00:00.123456+00" }, + { db } + ); + await upsertMirrorPage( + { mirrorId, entity: "tools", entityId, notionPageId: "n-2", sourceUpdatedAt: "2026-09-23 10:05:00.654321+00" }, + { db } + ); + + const found = await getMirrorPageIds(mirrorId, "tools", [entityId, crypto.randomUUID(), "not-a-uuid"], { db }); + expect(found).toEqual(new Map([[entityId, "n-2"]])); + + const [row] = await rawRows<{ source: string; n: number }>( + db, + sql`select (source_updated_at at time zone 'UTC')::text as source, + (select count(*)::int from mirror_pages where entity_id = ${entityId}) as n + from mirror_pages where entity_id = ${entityId}` + ); + // Written from text, so Postgres' microseconds survive (revision.ts). + expect(row.source).toMatch(/10:05:00\.654321/); + expect(Number(row.n)).toBe(1); + }); + + it("accepts a null source_updated_at", async () => { + const entityId = crypto.randomUUID(); + await upsertMirrorPage({ mirrorId, entity: "units", entityId, notionPageId: "n-u", sourceUpdatedAt: null }, { db }); + const [row] = await db.select().from(mirrorPages).where(eq(mirrorPages.entityId, entityId)); + expect(row.sourceUpdatedAt).toBeNull(); + expect(row.pushedAt).toBeInstanceOf(Date); + }); + + it("keeps each mirror's and each entity's pages apart", async () => { + const entityId = crypto.randomUUID(); + await upsertMirrorPage({ mirrorId, entity: "tools", entityId, notionPageId: "mine", sourceUpdatedAt: null }, { db }); + await upsertMirrorPage( + { mirrorId: otherMirrorId, entity: "tools", entityId, notionPageId: "theirs", sourceUpdatedAt: null }, + { db } + ); + expect((await getMirrorPageIds(mirrorId, "tools", [entityId], { db })).get(entityId)).toBe("mine"); + expect((await getMirrorPageIds(otherMirrorId, "tools", [entityId], { db })).get(entityId)).toBe("theirs"); + expect((await getMirrorPageIds(mirrorId, "units", [entityId], { db })).size).toBe(0); + expect((await getMirrorPageIds(mirrorId, "tools", [], { db })).size).toBe(0); + }); + + it("forgets one page", async () => { + const entityId = crypto.randomUUID(); + await upsertMirrorPage({ mirrorId, entity: "resources", entityId, notionPageId: "n-r", sourceUpdatedAt: null }, { db }); + await deleteMirrorPage(mirrorId, "resources", entityId, { db }); + expect((await getMirrorPageIds(mirrorId, "resources", [entityId], { db })).size).toBe(0); + }); + + it("maps maintenance to maintenance_logs", () => { + expect(MIRROR_SOURCE_TABLE.maintenance).toBe("maintenance_logs"); + }); + + it("lists pages whose source row is gone, per entity, oldest first and capped", async () => { + const [kept] = await db + .insert(tools) + .values({ name: "Kept", slug: `kept-${crypto.randomUUID()}` }) + .returning({ id: tools.id }); + const goneA = crypto.randomUUID(); + const goneB = crypto.randomUUID(); + for (const [entityId, page, ago] of [ + [kept.id, "n-kept", 3], + [goneA, "n-gone-a", 2], + [goneB, "n-gone-b", 1], + ] as const) { + await upsertMirrorPage({ mirrorId, entity: "tools", entityId, notionPageId: page, sourceUpdatedAt: null }, { db }); + await db.execute( + sql`update mirror_pages set pushed_at = now() - ${sql.raw(`interval '${ago + 100} minutes'`)} where entity_id = ${entityId}` + ); + } + + const orphans = await listOrphanedMirrorPages(mirrorId, "tools", { limit: 10 }, { db }); + expect(orphans.filter((row) => row.notionPageId.startsWith("n-gone") || row.notionPageId === "n-kept")).toEqual([ + { entityId: goneA, notionPageId: "n-gone-a" }, + { entityId: goneB, notionPageId: "n-gone-b" }, + ]); + expect(await listOrphanedMirrorPages(mirrorId, "tools", { limit: 1, db })).toEqual([ + { entityId: goneA, notionPageId: "n-gone-a" }, + ]); + }); + + it("anti-joins maintenance against maintenance_logs", async () => { + const [category] = await db.insert(categories).values({ name: "C", group: "G" }).returning({ id: categories.id }); + const [log] = await db + .insert(maintenanceLogs) + .values({ title: "Nozzle clog", type: "issue_report", priority: "low", status: "open" }) + .returning({ id: maintenanceLogs.id }); + const gone = crypto.randomUUID(); + await upsertMirrorPage({ mirrorId: otherMirrorId, entity: "maintenance", entityId: log.id, notionPageId: "n-log", sourceUpdatedAt: null }, { db }); + await upsertMirrorPage({ mirrorId: otherMirrorId, entity: "maintenance", entityId: gone, notionPageId: "n-gone", sourceUpdatedAt: null }, { db }); + await upsertMirrorPage({ mirrorId: otherMirrorId, entity: "categories", entityId: category.id, notionPageId: "n-cat", sourceUpdatedAt: null }, { db }); + + expect(await listOrphanedMirrorPages(otherMirrorId, "maintenance", {}, { db })).toEqual([ + { entityId: gone, notionPageId: "n-gone" }, + ]); + expect(await listOrphanedMirrorPages(otherMirrorId, "categories", {}, { db })).toEqual([]); + }); +}); diff --git a/v5/src/lib/data/mirror-pages.ts b/v5/src/lib/data/mirror-pages.ts new file mode 100644 index 0000000..b9eeab9 --- /dev/null +++ b/v5/src/lib/data/mirror-pages.ts @@ -0,0 +1,167 @@ +import { and, eq, inArray, sql } from "drizzle-orm"; +import { getDb } from "../db/client.ts"; +import { rawRows } from "../db/raw.ts"; +import { mirrorPages } from "../db/schema/mirror.ts"; +import { MIRROR_ENTITY, isOneOf, type MirrorEntity } from "../db/schema/vocabulary.ts"; +import type { Db } from "../db/types.ts"; +import { isUuid } from "./uuid.ts"; + +/** + * `mirror_pages` — which Notion page mirrors which app row, per mirror (spec + * §3.8 "Push" step 3, §4.12). + * + * The push looks a batch of rows up here, updates the pages it finds and + * creates the rest, recording each new page id as it goes, so a push cut short + * by its budget never creates the same page twice. + * + * `source_updated_at` is written from the text the push selected + * (`updated_at::text`) as `$::timestamptz`, never from a JavaScript `Date`, so + * it keeps Postgres' microseconds (see `revision.ts`). + * + * Relative imports with `.ts` extensions, no `@/` alias and no `server-only`: + * workflow step code loads this module. + */ + +export interface MirrorPageOptions { + /** A handle to use instead of {@link getDb} — tests pass an isolated one. */ + db?: Db; +} + +/** The Postgres table each entity mirrors. `maintenance` is `maintenance_logs`. */ +export const MIRROR_SOURCE_TABLE: Readonly> = { + categories: "categories", + locations: "locations", + tools: "tools", + units: "units", + resources: "resources", + maintenance: "maintenance_logs", + projects: "projects", +}; + +async function handle(options: MirrorPageOptions): Promise { + return options.db ?? (await getDb()); +} + +/** `entityId → notion_page_id` for the rows of `entityIds` that already have a page. */ +export async function getMirrorPageIds( + mirrorId: string, + entity: MirrorEntity, + entityIds: string[], + options: MirrorPageOptions = {} +): Promise> { + const ids = [...new Set(entityIds.filter(isUuid))]; + if (!isUuid(mirrorId) || ids.length === 0 || !isOneOf(MIRROR_ENTITY, entity)) return new Map(); + const db = await handle(options); + const rows = await db + .select({ entityId: mirrorPages.entityId, notionPageId: mirrorPages.notionPageId }) + .from(mirrorPages) + .where(and(eq(mirrorPages.mirrorId, mirrorId), eq(mirrorPages.entity, entity), inArray(mirrorPages.entityId, ids))); + return new Map(rows.map((row) => [row.entityId, row.notionPageId])); +} + +/** + * Record (or re-record) the page that mirrors one row, stamping `pushed_at = now()`. + * + * With `generation` (the push's claim), the row is written only while the + * mirror is still at that `mapping_generation`: a push still running after + * Create databases or Save mapping changed the mapping must not record pages + * in the old databases over the reset — the next push would take them as + * mirrored and never write the new ones. True when the row was written. + */ +export async function upsertMirrorPage( + input: { + mirrorId: string; + entity: MirrorEntity; + entityId: string; + notionPageId: string; + /** The source row's `updated_at::text`, straight from the push's SELECT. */ + sourceUpdatedAt: string | null; + /** The push's claimed `mapping_generation`; omitted, the write is unconditional. */ + generation?: number; + }, + options: MirrorPageOptions = {} +): Promise { + const db = await handle(options); + const sourceUpdatedAt = input.sourceUpdatedAt === null ? null : sql`${input.sourceUpdatedAt}::timestamptz`; + if (input.generation !== undefined) { + const rows = await rawRows<{ entity_id: string }>( + db, + sql` + insert into mirror_pages (mirror_id, entity, entity_id, notion_page_id, pushed_at, source_updated_at) + select ${input.mirrorId}::uuid, ${input.entity}, ${input.entityId}::uuid, ${input.notionPageId}, now(), ${sourceUpdatedAt} + where exists ( + select 1 from notion_mirrors n + where n.id = ${input.mirrorId}::uuid and n.mapping_generation = ${input.generation} + ) + on conflict (mirror_id, entity, entity_id) do update + set notion_page_id = excluded.notion_page_id, + pushed_at = excluded.pushed_at, + source_updated_at = excluded.source_updated_at + returning entity_id + ` + ); + return rows.length > 0; + } + await db + .insert(mirrorPages) + .values({ + mirrorId: input.mirrorId, + entity: input.entity, + entityId: input.entityId, + notionPageId: input.notionPageId, + pushedAt: sql`now()`, + sourceUpdatedAt, + }) + .onConflictDoUpdate({ + target: [mirrorPages.mirrorId, mirrorPages.entity, mirrorPages.entityId], + set: { notionPageId: input.notionPageId, pushedAt: sql`now()`, sourceUpdatedAt }, + }); + return true; +} + +/** Forget one row's page — after archiving it, or when Notion says it is gone. */ +export async function deleteMirrorPage( + mirrorId: string, + entity: MirrorEntity, + entityId: string, + options: MirrorPageOptions = {} +): Promise { + if (!isUuid(mirrorId) || !isUuid(entityId)) return; + const db = await handle(options); + await db + .delete(mirrorPages) + .where(and(eq(mirrorPages.mirrorId, mirrorId), eq(mirrorPages.entity, entity), eq(mirrorPages.entityId, entityId))); +} + +/** + * Pages whose source row no longer exists — an anti-join from `mirror_pages` + * to the entity's table — oldest push first, at most `limit` (100 by default). + * The push archives these pages and then forgets them. + * + * `db` is accepted in the third argument as well as the fourth, so both + * `(id, entity, { limit }, { db })` and `(id, entity, { limit, db })` work. + */ +export async function listOrphanedMirrorPages( + mirrorId: string, + entity: MirrorEntity, + query: { limit?: number; db?: Db } = {}, + options: MirrorPageOptions = {} +): Promise<{ entityId: string; notionPageId: string }[]> { + if (!isUuid(mirrorId) || !isOneOf(MIRROR_ENTITY, entity)) return []; + const limit = Math.max(1, Math.min(1000, Math.trunc(query.limit ?? 100))); + const db = options.db ?? query.db ?? (await getDb()); + const table = sql.raw(`"${MIRROR_SOURCE_TABLE[entity]}"`); + const rows = await rawRows<{ entity_id: string; notion_page_id: string }>( + db, + sql` + select mp.entity_id, mp.notion_page_id + from mirror_pages mp + where mp.mirror_id = ${mirrorId} + and mp.entity = ${entity} + and not exists (select 1 from ${table} s where s.id = mp.entity_id) + order by mp.pushed_at, mp.entity_id + limit ${limit} + ` + ); + return rows.map((row) => ({ entityId: row.entity_id, notionPageId: row.notion_page_id })); +} diff --git a/v5/src/lib/data/mirrors.test.ts b/v5/src/lib/data/mirrors.test.ts new file mode 100644 index 0000000..d28e1b6 --- /dev/null +++ b/v5/src/lib/data/mirrors.test.ts @@ -0,0 +1,693 @@ +// @vitest-environment node +import { eq, sql } from "drizzle-orm"; +import { createPgliteDb } from "../db/pglite"; +import { rawRows } from "../db/raw"; +import { categories, mirrorPages, notionMirrors, tools, user } from "../db/schema/index"; +import type { Db } from "../db/types"; +import { + claimCoalescedPush, + claimManualSync, + claimMirrorRun, + disconnectMirror, + finishMirrorRun, + getMirror, + getMirrorForOwner, + getMirrorTokenCiphertext, + getMirrorViewForOwner, + listMirrorsDueForBackstop, + normalizeMirrorMapping, + releaseCoalescedPush, + releaseManualSync, + releaseMirrorRun, + resetMirrorEntities, + saveMirrorConnection, + setMirrorMapping, + setMirrorPaused, + takeCoalescedPush, + type ClaimedMirror, +} from "./mirrors"; + +/** + * `notion_mirrors` against PGlite (spec §3.8, §4.12, §8). Every time window is + * staged with SQL relative to the database's own `now()`, the same clock the + * claims read, so nothing here depends on the test machine's clock agreeing + * with Postgres'. + */ + +const PAGE = "0f5e4a3c-1111-2222-3333-444455556666"; +const TOKEN_BYTES = new Uint8Array([1, 9, 9, 7, 42]); + +async function insertUser(db: Db): Promise { + const id = `u-${Math.random().toString(36).slice(2)}`; + await db.insert(user).values({ id, name: "Mirror Owner", email: `${id}@cornell.edu`, role: "admin" }); + return id; +} + +async function connected(db: Db, mapping: Record = { tools: "db-tools" }) { + const owner = await insertUser(db); + const { mirror } = await saveMirrorConnection( + { ownerUserId: owner, tokenCiphertext: TOKEN_BYTES, parentPageId: PAGE, parentPageTitle: "Mirror" }, + { db } + ); + if (Object.keys(mapping).length) await setMirrorMapping(mirror.id, mapping, { db }); + return { owner, id: mirror.id }; +} + +async function stage(db: Db, id: string, assignments: string): Promise { + await db.execute(sql`update notion_mirrors set ${sql.raw(assignments)} where id = ${id}`); +} + +function isClaimed(result: Awaited>): result is ClaimedMirror { + return !("skipped" in result); +} + +describe("mirror connection", () => { + let db: Db; + beforeAll(async () => { + db = await createPgliteDb(); + }); + + it("creates the owner's mirror, then reconnects it in place", async () => { + const owner = await insertUser(db); + const first = await saveMirrorConnection( + { ownerUserId: owner, tokenCiphertext: TOKEN_BYTES, parentPageId: PAGE, parentPageTitle: "Mirror" }, + { db } + ); + expect(first.created).toBe(true); + expect(first.mirror).toMatchObject({ ownerUserId: owner, hasToken: true, parentPageId: PAGE, mapping: {} }); + expect("tokenCiphertext" in first.mirror).toBe(false); + + const second = await saveMirrorConnection( + { ownerUserId: owner, tokenCiphertext: new Uint8Array([7]), parentPageId: PAGE, parentPageTitle: "Renamed" }, + { db } + ); + expect(second.created).toBe(false); + expect(second.mirror.id).toBe(first.mirror.id); + expect(second.mirror.parentPageTitle).toBe("Renamed"); + expect(Array.from((await getMirrorTokenCiphertext(first.mirror.id, { db }))!)).toEqual([7]); + }); + + it("reconnecting keeps the mapping and pages, resumes, and clears an error the old token caused", async () => { + const { owner, id } = await connected(db, { tools: "db-tools", units: "db-units" }); + const entityId = crypto.randomUUID(); + await db.insert(mirrorPages).values({ mirrorId: id, entity: "tools", entityId, notionPageId: "p-1" }); + await db + .update(notionMirrors) + .set({ + pausedAt: sql`now()`, + lastStatus: "failed", + lastError: { code: "unauthorized", entities: [], failed: 0, detail: "Notion 401 unauthorized" }, + }) + .where(eq(notionMirrors.id, id)); + + const { mirror } = await saveMirrorConnection( + { ownerUserId: owner, tokenCiphertext: TOKEN_BYTES, parentPageId: PAGE, parentPageTitle: null }, + { db } + ); + + expect(mirror.pausedAt).toBeNull(); + expect(mirror.lastError).toBeNull(); + expect(mirror.mapping).toEqual({ tools: "db-tools", units: "db-units" }); + const pages = await db.select().from(mirrorPages).where(eq(mirrorPages.mirrorId, id)); + expect(pages).toHaveLength(1); + }); + + it("reconnecting keeps an error the token did not cause", async () => { + const { owner, id } = await connected(db); + const error = { code: "database_not_found" as const, entities: ["tools" as const], failed: 0, detail: null }; + await db.update(notionMirrors).set({ lastError: error }).where(eq(notionMirrors.id, id)); + + const { mirror } = await saveMirrorConnection( + { ownerUserId: owner, tokenCiphertext: TOKEN_BYTES, parentPageId: PAGE, parentPageTitle: null }, + { db } + ); + expect(mirror.lastError).toEqual(error); + }); + + it("finds a mirror by owner and by id, and nothing for a malformed id", async () => { + const { owner, id } = await connected(db); + expect((await getMirrorForOwner(owner, { db }))?.id).toBe(id); + expect((await getMirror(id, { db }))?.ownerUserId).toBe(owner); + expect(await getMirror("not-a-uuid", { db })).toBeNull(); + expect(await getMirror(crypto.randomUUID(), { db })).toBeNull(); + expect(await getMirrorForOwner("nobody", { db })).toBeNull(); + expect(await getMirrorTokenCiphertext("not-a-uuid", { db })).toBeNull(); + }); + + it("disconnect forgets the token and the pending push, and keeps the mapping and pages", async () => { + const { owner, id } = await connected(db, { tools: "db-tools" }); + await db.insert(mirrorPages).values({ mirrorId: id, entity: "tools", entityId: crypto.randomUUID(), notionPageId: "p" }); + await stage(db, id, "push_requested_at = now()"); + + expect(await disconnectMirror(owner, { db })).toBe(true); + + const mirror = await getMirror(id, { db }); + expect(mirror?.hasToken).toBe(false); + expect(mirror?.pushRequestedAt).toBeNull(); + expect(mirror?.mapping).toEqual({ tools: "db-tools" }); + expect(await getMirrorTokenCiphertext(id, { db })).toBeNull(); + expect(await db.select().from(mirrorPages).where(eq(mirrorPages.mirrorId, id))).toHaveLength(1); + + expect(await disconnectMirror("nobody", { db })).toBe(false); + }); + + it("stores a normalised mapping", async () => { + const { id } = await connected(db, {}); + const mirror = await setMirrorMapping( + id, + { tools: " db-tools ", units: "", bogus: "db-x" } as unknown as Record, + { db } + ); + expect(mirror.mapping).toEqual({ tools: "db-tools" }); + await expect(setMirrorMapping(crypto.randomUUID(), {}, { db })).rejects.toThrow(/does not exist/); + }); + + it("normalizeMirrorMapping keeps only known entities with string ids", () => { + expect(normalizeMirrorMapping(null)).toEqual({}); + expect(normalizeMirrorMapping(["tools"])).toEqual({}); + expect(normalizeMirrorMapping({ projects: "p", tools: 3, maintenance: "m" })).toEqual({ maintenance: "m", projects: "p" }); + }); + + it("resetMirrorEntities clears only its entities' pages, and forces a full push", async () => { + const { id } = await connected(db); + const rows = (["tools", "units", "categories"] as const).map((entity) => ({ + mirrorId: id, + entity, + entityId: crypto.randomUUID(), + notionPageId: `p-${entity}`, + })); + await db.insert(mirrorPages).values(rows); + await stage(db, id, "last_synced_at = now()"); + + await resetMirrorEntities(id, ["tools", "units"], { db }); + + const left = await db.select({ entity: mirrorPages.entity }).from(mirrorPages).where(eq(mirrorPages.mirrorId, id)); + expect(left.map((row) => row.entity)).toEqual(["categories"]); + expect((await getMirror(id, { db }))?.lastSyncedAt).toBeNull(); + }); + + it("resetMirrorEntities marks the pages that link to the reset entities as not mirrored, and only those", async () => { + const { id } = await connected(db); + const entities = ["categories", "tools", "units", "resources", "maintenance", "projects"] as const; + await db.insert(mirrorPages).values( + entities.map((entity) => ({ + mirrorId: id, + entity, + entityId: crypto.randomUUID(), + notionPageId: `p-${entity}`, + sourceUpdatedAt: sql`now()`, + })) + ); + + // Tools recreated: every entity with a Tool / Tools relation must push again. + await resetMirrorEntities(id, ["tools"], { db }); + const afterTools = await rawRows<{ entity: string; stale: boolean }>( + db, + sql`select entity, source_updated_at is null as stale from mirror_pages where mirror_id = ${id} order by entity` + ); + expect(Object.fromEntries(afterTools.map((row) => [row.entity, row.stale]))).toEqual({ + categories: false, + maintenance: true, + projects: true, + resources: true, + units: true, + }); + + // Units recreated: only maintenance links to a unit. + await db.execute(sql`update mirror_pages set source_updated_at = now() where mirror_id = ${id}`); + await resetMirrorEntities(id, ["units"], { db }); + const afterUnits = await rawRows<{ entity: string; stale: boolean }>( + db, + sql`select entity, source_updated_at is null as stale from mirror_pages where mirror_id = ${id} order by entity` + ); + expect(Object.fromEntries(afterUnits.map((row) => [row.entity, row.stale]))).toEqual({ + categories: false, + maintenance: true, + projects: false, + resources: false, + }); + }); + + it("a mapping change or reset moves the generation; saving the same mapping does not", async () => { + const { id } = await connected(db, { tools: "db-tools" }); + const generation = async () => + Number((await rawRows<{ g: number }>(db, sql`select mapping_generation as g from notion_mirrors where id = ${id}`))[0].g); + const start = await generation(); + + await setMirrorMapping(id, { tools: "db-tools" }, { db }); + expect(await generation()).toBe(start); + await setMirrorMapping(id, { tools: "db-tools", units: "db-units" }, { db }); + expect(await generation()).toBe(start + 1); + await resetMirrorEntities(id, ["units"], { db }); + expect(await generation()).toBe(start + 2); + }); + + it("pauses and resumes, keeping the first paused_at", async () => { + const { owner, id } = await connected(db); + await stage(db, id, "paused_at = now() - interval '1 hour'"); + const before = (await getMirror(id, { db }))!.pausedAt!; + + const paused = await setMirrorPaused(owner, true, { db }); + expect(paused?.pausedAt?.getTime()).toBe(before.getTime()); + + const resumed = await setMirrorPaused(owner, false, { db }); + expect(resumed?.pausedAt).toBeNull(); + expect(await setMirrorPaused("nobody", true, { db })).toBeNull(); + }); +}); + +describe("claimMirrorRun and finishMirrorRun", () => { + let db: Db; + beforeAll(async () => { + db = await createPgliteDb(); + }); + + it("claims an idle, connected, unpaused mirror and hands over the ciphertext", async () => { + const { id } = await connected(db); + const result = await claimMirrorRun(id, { db }); + expect(isClaimed(result)).toBe(true); + if (!isClaimed(result)) return; + expect(result.id).toBe(id); + expect(result.runningSince).toBeInstanceOf(Date); + expect(Array.from(result.tokenCiphertext!)).toEqual(Array.from(TOKEN_BYTES)); + expect(result.since).toBeNull(); + }); + + it("skips a paused mirror", async () => { + const { id } = await connected(db); + await stage(db, id, "paused_at = now()"); + expect(await claimMirrorRun(id, { db })).toEqual({ skipped: "paused" }); + }); + + it("skips a disconnected mirror", async () => { + const { owner, id } = await connected(db); + await disconnectMirror(owner, { db }); + expect(await claimMirrorRun(id, { db })).toEqual({ skipped: "not_connected" }); + }); + + it("skips a mirror another push claimed less than 15 minutes ago", async () => { + const { id } = await connected(db); + expect(isClaimed(await claimMirrorRun(id, { db }))).toBe(true); + expect(await claimMirrorRun(id, { db })).toEqual({ skipped: "running" }); + + await stage(db, id, "running_since = now() - interval '14 minutes'"); + expect(await claimMirrorRun(id, { db })).toEqual({ skipped: "running" }); + }); + + it("takes over a push that has been running for more than 15 minutes", async () => { + const { id } = await connected(db); + await stage(db, id, "running_since = now() - interval '16 minutes'"); + expect(isClaimed(await claimMirrorRun(id, { db }))).toBe(true); + }); + + it("is not_found for an unknown or malformed id", async () => { + expect(await claimMirrorRun(crypto.randomUUID(), { db })).toEqual({ skipped: "not_found" }); + expect(await claimMirrorRun("nope", { db })).toEqual({ skipped: "not_found" }); + }); + + it("takes the watermark 5 minutes before the claim, in the same statement", async () => { + const { id } = await connected(db); + const result = await claimMirrorRun(id, { db }); + if (!isClaimed(result)) throw new Error("expected a claim"); + + const [check] = await rawRows<{ same: boolean }>( + db, + sql`select (${result.watermark}::timestamptz = running_since - interval '5 minutes') as same from notion_mirrors where id = ${id}` + ); + expect(check.same).toBe(true); + }); + + it("carries last_synced_at as text with its microseconds, and advances to text the same way", async () => { + const { id } = await connected(db); + await stage(db, id, "last_synced_at = '2026-09-23 10:00:00.123456+00'"); + + const result = await claimMirrorRun(id, { db }); + if (!isClaimed(result)) throw new Error("expected a claim"); + expect(result.since).toMatch(/\.123456/); + // Whatever the session's TimeZone, the text names the same instant. + const [same] = await rawRows<{ same: boolean }>( + db, + sql`select (${result.since}::timestamptz = '2026-09-23 10:00:00.123456+00'::timestamptz) as same` + ); + expect(same.same).toBe(true); + + await finishMirrorRun( + id, + { status: "ok", error: null, advanceTo: "2026-09-23 11:00:00.654321+00", pause: false, generation: result.generation }, + { db } + ); + const [row] = await rawRows<{ synced: string }>( + db, + sql`select (last_synced_at at time zone 'UTC')::text as synced from notion_mirrors where id = ${id}` + ); + expect(row.synced).toMatch(/11:00:00\.654321/); + }); + + it("finishing clears the guard, stamps last_run_at and records the result", async () => { + const { id } = await connected(db); + await claimMirrorRun(id, { db }); + const error = { code: "rows_failed" as const, entities: ["tools" as const], failed: 2, detail: "Notion 400 validation_error" }; + + await finishMirrorRun(id, { status: "partial", error, advanceTo: null, pause: false, generation: 1 }, { db }); + + const mirror = (await getMirror(id, { db }))!; + expect(mirror.runningSince).toBeNull(); + expect(mirror.lastRunAt).toBeInstanceOf(Date); + expect(mirror.lastStatus).toBe("partial"); + expect(mirror.lastError).toEqual(error); + expect(mirror.pausedAt).toBeNull(); + // A partial push does not advance: the failed rows are selected again. + expect(mirror.lastSyncedAt).toBeNull(); + expect(isClaimed(await claimMirrorRun(id, { db }))).toBe(true); + }); + + it("keeps last_synced_at when advanceTo is null, and pauses when asked", async () => { + const { id } = await connected(db); + await stage(db, id, "last_synced_at = '2026-09-01 00:00:00.000001+00'"); + await claimMirrorRun(id, { db }); + + await finishMirrorRun( + id, + { + status: "failed", + error: { code: "unauthorized", entities: [], failed: 0, detail: null }, + advanceTo: null, + pause: true, + generation: 1, + }, + { db } + ); + + const [row] = await rawRows<{ synced: string; paused: boolean }>( + db, + sql`select (last_synced_at at time zone 'UTC')::text as synced, paused_at is not null as paused from notion_mirrors where id = ${id}` + ); + expect(row.synced).toMatch(/2026-09-01 00:00:00\.000001/); + expect(row.paused).toBe(true); + expect(await claimMirrorRun(id, { db })).toEqual({ skipped: "paused" }); + }); + + it("does not advance last_synced_at over a reset that landed during the push", async () => { + const { id } = await connected(db); + await stage(db, id, "last_synced_at = '2026-09-01 00:00:00+00'"); + const claim = await claimMirrorRun(id, { db }); + if (!isClaimed(claim)) throw new Error("expected a claim"); + + // Create databases made the categories database while the push ran. + await resetMirrorEntities(id, ["categories"], { db }); + const finished = await finishMirrorRun( + id, + { status: "ok", error: null, advanceTo: claim.watermark, pause: false, generation: claim.generation }, + { db } + ); + + expect(finished).toEqual({ current: false }); + const mirror = (await getMirror(id, { db }))!; + expect(mirror.lastSyncedAt).toBeNull(); + expect(mirror.runningSince).toBeNull(); + }); + + it("does not advance last_synced_at over a mapping change during the push, even from a first sync", async () => { + const { id } = await connected(db, { tools: "db-tools" }); + const claim = await claimMirrorRun(id, { db }); + if (!isClaimed(claim)) throw new Error("expected a claim"); + expect(claim.since).toBeNull(); + + await setMirrorMapping(id, { tools: "db-tools", categories: "db-categories" }, { db }); + const finished = await finishMirrorRun( + id, + { status: "ok", error: null, advanceTo: claim.watermark, pause: false, generation: claim.generation }, + { db } + ); + + expect(finished).toEqual({ current: false }); + expect((await getMirror(id, { db }))!.lastSyncedAt).toBeNull(); + }); + + it("advances when nothing changed the mapping during the push", async () => { + const { id } = await connected(db); + const claim = await claimMirrorRun(id, { db }); + if (!isClaimed(claim)) throw new Error("expected a claim"); + const finished = await finishMirrorRun( + id, + { status: "ok", error: null, advanceTo: claim.watermark, pause: false, generation: claim.generation }, + { db } + ); + expect(finished).toEqual({ current: true }); + expect((await getMirror(id, { db }))!.lastSyncedAt).toBeInstanceOf(Date); + }); + + it("skips the mirror of an owner who was demoted or banned", async () => { + const demoted = await connected(db); + await db.update(user).set({ role: "user" }).where(eq(user.id, demoted.owner)); + expect(await claimMirrorRun(demoted.id, { db })).toEqual({ skipped: "owner_not_allowed" }); + + const banned = await connected(db); + await db.update(user).set({ banned: true }).where(eq(user.id, banned.owner)); + expect(await claimMirrorRun(banned.id, { db })).toEqual({ skipped: "owner_not_allowed" }); + + // Promoted back, or unbanned: the mirror pushes again. + await db.update(user).set({ role: "super_admin" }).where(eq(user.id, demoted.owner)); + expect(isClaimed(await claimMirrorRun(demoted.id, { db }))).toBe(true); + }); + + it("releasing a run frees the guard and stamps last_run_at without touching the result", async () => { + const { id } = await connected(db); + const error = { code: "rows_failed" as const, entities: ["tools" as const], failed: 1, detail: null }; + await claimMirrorRun(id, { db }); + await finishMirrorRun(id, { status: "partial", error, advanceTo: null, pause: false, generation: 1 }, { db }); + await stage(db, id, "last_run_at = now() - interval '1 hour'"); + await claimMirrorRun(id, { db }); + + await releaseMirrorRun(id, { db }); + + const [row] = await rawRows<{ recent: boolean }>( + db, + sql`select last_run_at > now() - interval '1 minute' as recent from notion_mirrors where id = ${id}` + ); + expect(row.recent).toBe(true); + const mirror = (await getMirror(id, { db }))!; + expect(mirror.runningSince).toBeNull(); + expect(mirror.lastStatus).toBe("partial"); + expect(mirror.lastError).toEqual(error); + expect(isClaimed(await claimMirrorRun(id, { db }))).toBe(true); + }); +}); + +describe("Sync now", () => { + let db: Db; + beforeAll(async () => { + db = await createPgliteDb(); + }); + + it("claims once, then refuses for 15 minutes with the seconds left", async () => { + const { owner, id } = await connected(db); + expect(await claimManualSync(owner, { db })).toEqual({ ok: true, mirrorId: id }); + + const again = await claimManualSync(owner, { db }); + expect(again.ok).toBe(false); + if (again.ok) return; + expect(again.reason).toBe("too_soon"); + expect(again.retryAfterSeconds).toBeGreaterThan(890); + expect(again.retryAfterSeconds).toBeLessThanOrEqual(900); + + await stage(db, id, "sync_requested_at = now() - interval '14 minutes'"); + const later = await claimManualSync(owner, { db }); + expect(later).toMatchObject({ ok: false, reason: "too_soon" }); + if (!later.ok) expect(later.retryAfterSeconds).toBeLessThanOrEqual(60); + + await stage(db, id, "sync_requested_at = now() - interval '16 minutes'"); + expect(await claimManualSync(owner, { db })).toEqual({ ok: true, mirrorId: id }); + }); + + it("gives the claim back when the push could not start", async () => { + const { owner, id } = await connected(db); + await claimManualSync(owner, { db }); + await releaseManualSync(id, { db }); + expect(await claimManualSync(owner, { db })).toEqual({ ok: true, mirrorId: id }); + }); + + it("refuses with the reason: not found, not connected, paused, not mapped", async () => { + expect(await claimManualSync("nobody", { db })).toEqual({ ok: false, reason: "not_found" }); + + const disconnected = await connected(db); + await disconnectMirror(disconnected.owner, { db }); + expect(await claimManualSync(disconnected.owner, { db })).toEqual({ ok: false, reason: "not_connected" }); + + const paused = await connected(db); + await setMirrorPaused(paused.owner, true, { db }); + expect(await claimManualSync(paused.owner, { db })).toEqual({ ok: false, reason: "paused" }); + + const unmapped = await connected(db, {}); + expect(await claimManualSync(unmapped.owner, { db })).toEqual({ ok: false, reason: "not_mapped" }); + }); + + it("refuses while another push holds the mirror, without spending the window", async () => { + const { owner, id } = await connected(db); + await stage(db, id, "running_since = now()"); + + expect(await claimManualSync(owner, { db })).toEqual({ ok: false, reason: "running" }); + expect((await getMirror(id, { db }))!.syncRequestedAt).toBeNull(); + + // A push that died long ago does not hold it. + await stage(db, id, "running_since = now() - interval '16 minutes'"); + expect(await claimManualSync(owner, { db })).toEqual({ ok: true, mirrorId: id }); + }); +}); + +describe("coalesced pushes", () => { + let db: Db; + beforeAll(async () => { + db = await createPgliteDb(); + }); + + it("claims every active mirror once, releases, reclaims when stale, and takes", async () => { + const a = await connected(db); + const b = await connected(db); + const paused = await connected(db); + await setMirrorPaused(paused.owner, true, { db }); + const gone = await connected(db); + await disconnectMirror(gone.owner, { db }); + + expect((await claimCoalescedPush({ db })).sort()).toEqual([a.id, b.id].sort()); + // A push is already on its way: a second change claims nothing. + expect(await claimCoalescedPush({ db })).toEqual([]); + + await releaseCoalescedPush([a.id], { db }); + expect(await claimCoalescedPush({ db })).toEqual([a.id]); + + // A claim older than 10 minutes is a coalescing run that never finished. + await stage(db, b.id, "push_requested_at = now() - interval '11 minutes'"); + expect(await claimCoalescedPush({ db })).toEqual([b.id]); + + // The run wakes: it clears every claim and pushes the mirrors still active. + await stage(db, paused.id, "push_requested_at = now()"); + expect((await takeCoalescedPush({ db })).sort()).toEqual([a.id, b.id].sort()); + const [left] = await rawRows<{ n: number }>( + db, + sql`select count(*)::int as n from notion_mirrors where push_requested_at is not null` + ); + expect(Number(left.n)).toBe(0); + expect(await takeCoalescedPush({ db })).toEqual([]); + expect((await claimCoalescedPush({ db })).sort()).toEqual([a.id, b.id].sort()); + }); + + it("claims and takes nothing for a demoted or banned owner", async () => { + const demoted = await connected(db); + await db.update(user).set({ role: "user" }).where(eq(user.id, demoted.owner)); + const banned = await connected(db); + await db.update(user).set({ banned: true }).where(eq(user.id, banned.owner)); + + const claimed = await claimCoalescedPush({ db }); + expect(claimed).not.toContain(demoted.id); + expect(claimed).not.toContain(banned.id); + + await stage(db, demoted.id, "push_requested_at = now()"); + expect(await takeCoalescedPush({ db })).not.toContain(demoted.id); + }); +}); + +describe("listMirrorsDueForBackstop", () => { + let db: Db; + beforeAll(async () => { + db = await createPgliteDb(); + }); + + it("lists active, idle mirrors that are behind, and only those", async () => { + const [category] = await db + .insert(categories) + .values({ name: "Backstop", group: "Test", updatedAt: sql`now() - interval '2 hours'` }) + .returning({ id: categories.id }); + await db.insert(tools).values({ name: "Old tool", slug: `old-${crypto.randomUUID()}`, updatedAt: sql`now() - interval '2 hours'` }); + + const neverSynced = await connected(db); + const upToDate = await connected(db); + await stage(db, upToDate.id, "last_synced_at = now() - interval '1 hour', last_status = 'ok'"); + const partial = await connected(db); + await stage(db, partial.id, "last_synced_at = now() - interval '1 hour', last_status = 'partial'"); + const running = await connected(db); + await stage(db, running.id, "running_since = now()"); + const paused = await connected(db); + await setMirrorPaused(paused.owner, true, { db }); + const disconnected = await connected(db); + await disconnectMirror(disconnected.owner, { db }); + const demoted = await connected(db); + await db.update(user).set({ role: "user" }).where(eq(user.id, demoted.owner)); + const banned = await connected(db); + await db.update(user).set({ banned: true }).where(eq(user.id, banned.owner)); + + expect((await listMirrorsDueForBackstop({ db })).sort()).toEqual([neverSynced.id, partial.id].sort()); + + // A category edited after the last sync puts the up-to-date mirror behind. + await db.update(categories).set({ name: "Backstop, renamed" }).where(eq(categories.id, category.id)); + expect((await listMirrorsDueForBackstop({ db })).sort()).toEqual( + [neverSynced.id, partial.id, upToDate.id].sort() + ); + }); +}); + +describe("getMirrorViewForOwner", () => { + let db: Db; + beforeAll(async () => { + db = await createPgliteDb(); + }); + + it("is null without a mirror", async () => { + expect(await getMirrorViewForOwner("nobody", { db })).toBeNull(); + }); + + it("renders a fresh mirror with nothing pending and Sync now available", async () => { + const { owner, id } = await connected(db, { tools: "db-tools" }); + const view = await getMirrorViewForOwner(owner, { db }); + expect(view).toEqual({ + id, + connected: true, + parentPageId: PAGE, + parentPageTitle: "Mirror", + mapping: { tools: "db-tools" }, + paused: false, + running: false, + syncPending: false, + pushScheduled: false, + lastSyncedAt: null, + lastRunAt: null, + lastStatus: null, + lastError: null, + syncAvailableAt: null, + }); + expect(JSON.stringify(view)).not.toMatch(/token/i); + }); + + it("computes running, syncPending, pushScheduled and syncAvailableAt in SQL, as ISO strings", async () => { + const { owner, id } = await connected(db); + await stage( + db, + id, + `running_since = now(), sync_requested_at = '2099-01-01 00:00:00+00', push_requested_at = now(), + last_synced_at = '2026-09-23 10:00:00.5+00', last_run_at = '2026-09-23 10:01:00+00', last_status = 'ok', paused_at = now()` + ); + + const view = (await getMirrorViewForOwner(owner, { db }))!; + expect(view.running).toBe(true); + expect(view.syncPending).toBe(true); + expect(view.pushScheduled).toBe(true); + expect(view.paused).toBe(true); + expect(view.lastStatus).toBe("ok"); + expect(view.lastSyncedAt).toBe("2026-09-23T10:00:00.500Z"); + expect(view.lastRunAt).toBe("2026-09-23T10:01:00.000Z"); + expect(view.syncAvailableAt).toBe("2099-01-01T00:15:00.000Z"); + }); + + it("treats a stale running_since as not running and an old Sync now as settled", async () => { + const { owner, id } = await connected(db); + await stage( + db, + id, + "running_since = now() - interval '16 minutes', sync_requested_at = now() - interval '20 minutes', last_run_at = now() - interval '19 minutes'" + ); + const view = (await getMirrorViewForOwner(owner, { db }))!; + expect(view.running).toBe(false); + expect(view.syncPending).toBe(false); + expect(view.syncAvailableAt).toBeNull(); + }); +}); diff --git a/v5/src/lib/data/mirrors.ts b/v5/src/lib/data/mirrors.ts new file mode 100644 index 0000000..b2ef719 --- /dev/null +++ b/v5/src/lib/data/mirrors.ts @@ -0,0 +1,713 @@ +import { and, eq, inArray, sql, type SQL } from "drizzle-orm"; +import { getDb } from "../db/client.ts"; +import { notionMirrors } from "../db/schema/mirror.ts"; +import { MIRROR_ENTITY, MIRROR_STATUS, isOneOf, type MirrorEntity, type MirrorStatus } from "../db/schema/vocabulary.ts"; +import type { Db } from "../db/types.ts"; +import { + MIRROR_COALESCE_STALE_MINUTES, + MIRROR_RUN_STALE_MINUTES, + MIRROR_SYNC_NOW_MINUTES, + MIRROR_WATERMARK_SAFETY_MINUTES, +} from "../mirror/limits.ts"; +import { relationDependents } from "../mirror/database-schemas.ts"; +import { MIRROR_OWNER_ROLES } from "../mirror/owner-roles.ts"; +import type { MirrorLastError, MirrorMapping, MirrorView } from "../mirror/types.ts"; +import { isUuid } from "./uuid.ts"; + +/** + * `notion_mirrors` — one admin's Notion mirror and the claims that keep its + * pushes from overlapping (spec §3.8, §4.12, §8). + * + * **Every claim is one conditional `UPDATE … RETURNING`** whose WHERE clause + * carries the state it moves from, so two callers racing for the same mirror + * cannot both win — the `pending-tools.ts` idiom. When a claim matches nothing, + * a follow-up read says why; the claim itself is still one statement. + * + * **Every time window is computed by Postgres, and no JavaScript `Date` ever + * reaches a comparison** (read `revision.ts`): `now()` has microseconds and a + * `Date` milliseconds, so a round-tripped timestamp would compare wrong on Neon + * and right on PGlite. The push watermark and `last_synced_at` therefore + * travel as `timestamptz::text` and go back in as `$::timestamptz`. + * + * **Nothing here returns the token** except {@link claimMirrorRun} and + * {@link getMirrorTokenCiphertext}, which hand the ciphertext to the push and to + * setup. {@link getMirrorViewForOwner} does not even select it. + * + * Relative imports with `.ts` extensions, no `@/` alias and no `server-only`: + * the mirror's workflow steps load this module from an esbuild bundle. + */ + +// ── Shapes ────────────────────────────────────────────────────────── + +export interface MirrorDataOptions { + /** A handle to use instead of {@link getDb} — tests pass an isolated one. */ + db?: Db; +} + +/** One mirror, minus its token. */ +export interface MirrorRecord { + id: string; + ownerUserId: string; + /** A token is stored; false after Disconnect. */ + hasToken: boolean; + parentPageId: string; + parentPageTitle: string | null; + mapping: MirrorMapping; + pausedAt: Date | null; + runningSince: Date | null; + pushRequestedAt: Date | null; + syncRequestedAt: Date | null; + lastSyncedAt: Date | null; + lastRunAt: Date | null; + lastStatus: MirrorStatus | null; + lastError: MirrorLastError | null; + createdAt: Date; + updatedAt: Date; +} + +/** What a push works from: the record, the ciphertext, and its two timestamps as text. */ +export interface ClaimedMirror extends MirrorRecord { + tokenCiphertext: Uint8Array | null; + /** + * Claim time minus {@link MIRROR_WATERMARK_SAFETY_MINUTES}, as Postgres' + * `timestamptz::text` (microseconds kept). Hand it to `finishMirrorRun` as + * `advanceTo` when everything up to it was pushed. + */ + watermark: string; + /** `last_synced_at::text`: push rows whose `updated_at > $since::timestamptz`. Null = push everything. */ + since: string | null; + /** + * `mapping_generation` at the claim. Hand it to `finishMirrorRun` and + * `upsertMirrorPage`: once the mapping changes under a running push, that + * push neither advances `last_synced_at` nor records a page. + */ + generation: number; +} + +export type ClaimMirrorRunResult = + | ClaimedMirror + | { skipped: "not_found" | "paused" | "not_connected" | "owner_not_allowed" | "running" }; + +export type ClaimManualSyncResult = + | { ok: true; mirrorId: string } + | { + ok: false; + reason: "not_found" | "not_connected" | "paused" | "not_mapped" | "too_soon" | "running"; + retryAfterSeconds?: number; + }; + +export interface FinishMirrorRunInput { + status: MirrorStatus; + error: MirrorLastError | null; + /** The claim's `watermark`, or null to leave `last_synced_at` where it is. */ + advanceTo: string | null; + /** + * The claim's `generation`. `advanceTo` is applied only while the mirror is + * still at it: a mapping changed or reset during the push (Create databases, + * Save mapping) forced a full push, and this push must not undo that. + */ + generation: number; + /** Pause the mirror (a revoked token, §5.8). Never un-pauses. */ + pause: boolean; +} + +// ── SQL pieces ────────────────────────────────────────────────────── + +/** `interval 'N minutes'` from a constant — rendered inline, never a parameter. */ +function minutes(n: number): SQL { + return sql.raw(`interval '${Math.trunc(n)} minutes'`); +} + +const m = notionMirrors; + +/** It has a token and is not paused. */ +const CONNECTED = sql`(${m.tokenCiphertext} is not null and ${m.pausedAt} is null)`; + +/** + * Its owner may still manage a mirror: their row holds a role that grants + * `mirror.manage` and is not banned (`mirror/owner-roles.ts`). A demoted or + * banned admin's mirror stops receiving pushes — it carries names and emails, + * and they could no longer pause it themselves. + */ +const OWNER_ALLOWED = sql`exists (select 1 from "user" as u where u.id = ${m.ownerUserId} and u.role in (${sql.join( + MIRROR_OWNER_ROLES.map((role) => sql`${role}`), + sql`, ` +)}) and u.banned is not true)`; + +/** A mirror is active when it is connected, not paused, and its owner may still manage it. */ +const ACTIVE = sql`(${CONNECTED} and ${OWNER_ALLOWED})`; + +/** Nobody is pushing it: no `running_since`, or one old enough to be a dead run. */ +const NOT_RUNNING = sql`(${m.runningSince} is null or ${m.runningSince} < now() - ${minutes(MIRROR_RUN_STALE_MINUTES)})`; + +/** At least one entity is mapped to a database id. */ +const MAPPED = sql`exists (select 1 from jsonb_each(${m.mapping}) as e where jsonb_typeof(e.value) = 'string' and e.value #>> '{}' <> '')`; + +/** Sync now is allowed: never pressed, or pressed longer ago than the window. */ +const SYNC_WINDOW_OPEN = sql`(${m.syncRequestedAt} is null or ${m.syncRequestedAt} < now() - ${minutes(MIRROR_SYNC_NOW_MINUTES)})`; + +/** A timestamptz as an ISO-8601 UTC string, milliseconds, computed by Postgres. */ +function isoText(expression: SQL): SQL { + return sql`to_char((${expression}) at time zone 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"')`; +} + +const RECORD_COLUMNS = { + id: m.id, + ownerUserId: m.ownerUserId, + hasToken: sql`(${m.tokenCiphertext} is not null)`, + parentPageId: m.parentPageId, + parentPageTitle: m.parentPageTitle, + mapping: m.mapping, + pausedAt: m.pausedAt, + runningSince: m.runningSince, + pushRequestedAt: m.pushRequestedAt, + syncRequestedAt: m.syncRequestedAt, + lastSyncedAt: m.lastSyncedAt, + lastRunAt: m.lastRunAt, + lastStatus: m.lastStatus, + lastError: m.lastError, + createdAt: m.createdAt, + updatedAt: m.updatedAt, +}; + +type RecordRow = { + id: string; + ownerUserId: string; + hasToken: boolean | string | null; + parentPageId: string; + parentPageTitle: string | null; + mapping: MirrorMapping | null; + pausedAt: Date | null; + runningSince: Date | null; + pushRequestedAt: Date | null; + syncRequestedAt: Date | null; + lastSyncedAt: Date | null; + lastRunAt: Date | null; + lastStatus: string | null; + lastError: MirrorLastError | null; + createdAt: Date; + updatedAt: Date; +}; + +function toRecord(row: RecordRow): MirrorRecord { + return { + id: row.id, + ownerUserId: row.ownerUserId, + hasToken: bool(row.hasToken), + parentPageId: row.parentPageId, + parentPageTitle: row.parentPageTitle, + mapping: normalizeMirrorMapping(row.mapping), + pausedAt: row.pausedAt, + runningSince: row.runningSince, + pushRequestedAt: row.pushRequestedAt, + syncRequestedAt: row.syncRequestedAt, + lastSyncedAt: row.lastSyncedAt, + lastRunAt: row.lastRunAt, + lastStatus: isOneOf(MIRROR_STATUS, row.lastStatus) ? row.lastStatus : null, + lastError: row.lastError ?? null, + createdAt: row.createdAt, + updatedAt: row.updatedAt, + }; +} + +/** Drivers agree on `boolean`, but a raw expression is cheap to be sure of. */ +function bool(value: unknown): boolean { + return value === true || value === "t" || value === "true"; +} + +/** + * Only known entities with a non-empty string id, in dependency order. The + * mapping is jsonb and was written by this app, but a hand-edited row should + * still read as something the push can trust. + */ +export function normalizeMirrorMapping(value: unknown): MirrorMapping { + const out: MirrorMapping = {}; + if (!value || typeof value !== "object" || Array.isArray(value)) return out; + const record = value as Record; + for (const entity of MIRROR_ENTITY) { + const id = record[entity]; + if (typeof id === "string" && id.trim()) out[entity] = id.trim(); + } + return out; +} + +async function handle(options: MirrorDataOptions): Promise { + return options.db ?? (await getDb()); +} + +// ── Reading ───────────────────────────────────────────────────────── + +/** One mirror by id, or null — including for anything not uuid-shaped. */ +export async function getMirror(id: string, options: MirrorDataOptions = {}): Promise { + if (!isUuid(id)) return null; + const db = await handle(options); + const [row] = await db.select(RECORD_COLUMNS).from(m).where(eq(m.id, id)).limit(1); + return row ? toRecord(row) : null; +} + +/** The signed-in admin's mirror, or null. Always found by owner — never by an id from the client (§8). */ +export async function getMirrorForOwner( + ownerUserId: string, + options: MirrorDataOptions = {} +): Promise { + const db = await handle(options); + const [row] = await db.select(RECORD_COLUMNS).from(m).where(eq(m.ownerUserId, ownerUserId)).limit(1); + return row ? toRecord(row) : null; +} + +/** + * What `/admin/mirror` renders. Every flag is computed by Postgres against its + * own `now()` — the same clock the claims use — and every date comes back as + * an ISO string. The token column is not selected. + */ +export async function getMirrorViewForOwner( + ownerUserId: string, + options: MirrorDataOptions = {} +): Promise { + const db = await handle(options); + const [row] = await db + .select({ + id: m.id, + connected: sql`(${m.tokenCiphertext} is not null)`, + parentPageId: m.parentPageId, + parentPageTitle: m.parentPageTitle, + mapping: m.mapping, + paused: sql`(${m.pausedAt} is not null)`, + running: sql`(${m.runningSince} is not null and ${m.runningSince} >= now() - ${minutes(MIRROR_RUN_STALE_MINUTES)})`, + syncPending: sql`(${m.syncRequestedAt} is not null and (${m.lastRunAt} is null or ${m.lastRunAt} < ${m.syncRequestedAt}))`, + pushScheduled: sql`(${m.pushRequestedAt} is not null)`, + lastSyncedAt: isoText(sql`${m.lastSyncedAt}`), + lastRunAt: isoText(sql`${m.lastRunAt}`), + lastStatus: m.lastStatus, + lastError: m.lastError, + syncAvailableAt: sql`case when ${SYNC_WINDOW_OPEN} then null else ${isoText( + sql`${m.syncRequestedAt} + ${minutes(MIRROR_SYNC_NOW_MINUTES)}` + )} end`, + }) + .from(m) + .where(eq(m.ownerUserId, ownerUserId)) + .limit(1); + if (!row) return null; + return { + id: row.id, + connected: bool(row.connected), + parentPageId: row.parentPageId, + parentPageTitle: row.parentPageTitle, + mapping: normalizeMirrorMapping(row.mapping), + paused: bool(row.paused), + running: bool(row.running), + syncPending: bool(row.syncPending), + pushScheduled: bool(row.pushScheduled), + lastSyncedAt: row.lastSyncedAt ?? null, + lastRunAt: row.lastRunAt ?? null, + lastStatus: isOneOf(MIRROR_STATUS, row.lastStatus) ? row.lastStatus : null, + lastError: row.lastError ?? null, + syncAvailableAt: row.syncAvailableAt ?? null, + }; +} + +/** The stored ciphertext, for setup calls (create databases, validate a mapping). Null when disconnected. */ +export async function getMirrorTokenCiphertext( + mirrorId: string, + options: MirrorDataOptions = {} +): Promise { + if (!isUuid(mirrorId)) return null; + const db = await handle(options); + const [row] = await db.select({ token: m.tokenCiphertext }).from(m).where(eq(m.id, mirrorId)).limit(1); + return row?.token ?? null; +} + +// ── Setup ─────────────────────────────────────────────────────────── + +/** + * Connect (or reconnect) the owner's mirror: an upsert on the owner. + * + * Reconnecting keeps the mapping and every `mirror_pages` row, so the same + * Notion pages are updated rather than duplicated. It clears `paused_at` — a + * new token is the answer to the 401 that paused it (§5.8) — and clears a + * `last_error` that the old token caused (`unauthorized`, `token_unreadable`, + * `key_unavailable`); any other error is still true and stays on the page. + * + * The caller has already validated the token with one read (§8) and encrypted + * it; this function never sees the plaintext. + */ +export async function saveMirrorConnection( + input: { + ownerUserId: string; + tokenCiphertext: Uint8Array; + parentPageId: string; + parentPageTitle: string | null; + }, + options: MirrorDataOptions = {} +): Promise<{ mirror: MirrorRecord; created: boolean }> { + const db = await handle(options); + const [row] = await db + .insert(m) + .values({ + ownerUserId: input.ownerUserId, + tokenCiphertext: input.tokenCiphertext, + parentPageId: input.parentPageId, + parentPageTitle: input.parentPageTitle, + }) + .onConflictDoUpdate({ + target: m.ownerUserId, + set: { + tokenCiphertext: input.tokenCiphertext, + parentPageId: input.parentPageId, + parentPageTitle: input.parentPageTitle, + pausedAt: null, + lastError: sql`case when ${m.lastError} ->> 'code' in ('unauthorized', 'token_unreadable', 'key_unavailable') then null else ${m.lastError} end`, + }, + }) + .returning({ ...RECORD_COLUMNS, created: sql`(xmax = 0)` }); + const { created, ...record } = row; + return { mirror: toRecord(record), created: bool(created) }; +} + +/** + * Forget the token (§3.8 "Disconnect"). The mapping and `mirror_pages` stay, + * so reconnecting picks up where it left off; a pending coalesced push is + * dropped, since it could not run. True when the owner had a mirror. + */ +export async function disconnectMirror(ownerUserId: string, options: MirrorDataOptions = {}): Promise { + const db = await handle(options); + const rows = await db + .update(m) + .set({ tokenCiphertext: null, pushRequestedAt: null }) + .where(eq(m.ownerUserId, ownerUserId)) + .returning({ id: m.id }); + return rows.length > 0; +} + +/** + * Replace the mapping. The caller validated every id (`applyPastedMapping`) or + * created the databases (`ensureMirrorDatabases`); unknown entities and empty + * ids are dropped here. + * + * @throws when the mirror does not exist — callers only ever hold an id they + * just read by owner. + */ +export async function setMirrorMapping( + mirrorId: string, + mapping: MirrorMapping, + options: MirrorDataOptions = {} +): Promise { + if (!isUuid(mirrorId)) throw new Error("setMirrorMapping: the mirror does not exist"); + const db = await handle(options); + const next = normalizeMirrorMapping(mapping); + const [row] = await db + .update(m) + .set({ + mapping: next, + // A mapping that changes moves the generation, so a push running under + // the old one records nothing more (see `finishMirrorRun`). Saving the + // same mapping again leaves a running push alone. + mappingGeneration: sql`case when ${m.mapping} = ${JSON.stringify(next)}::jsonb then ${m.mappingGeneration} else ${m.mappingGeneration} + 1 end`, + }) + .where(eq(m.id, mirrorId)) + .returning(RECORD_COLUMNS); + if (!row) throw new Error("setMirrorMapping: the mirror does not exist"); + return toRecord(row); +} + +/** + * Forget which pages mirror `entities` and make the next push a full one + * (`last_synced_at = null`) — what recreating a deleted database needs, since + * the old page ids point into a database that is gone (§5.8). + * + * Other entities' pages stay, but the **dependents** — every entity with a + * relation to one of `entities` (`relationDependents`: units, resources, + * maintenance and projects for tools; tools for categories and locations; + * maintenance for units) — are marked not mirrored (`source_updated_at = + * null`). Their relations point at the forgotten pages, and without the mark + * the push's own filter would skip them, since each is recorded at its current + * revision. The full push then updates them in place with the new links. + * + * Bumps `mapping_generation`, so a push already running under the old mapping + * cannot advance `last_synced_at` over this reset or record pages again. + */ +export async function resetMirrorEntities( + mirrorId: string, + entities: MirrorEntity[], + options: MirrorDataOptions = {} +): Promise { + const known = entities.filter((entity) => isOneOf(MIRROR_ENTITY, entity)); + if (!isUuid(mirrorId) || known.length === 0) return; + const db = await handle(options); + await db.transaction(async (tx) => { + await tx.execute( + sql`delete from mirror_pages where mirror_id = ${mirrorId} and entity in (${sql.join( + known.map((entity) => sql`${entity}`), + sql`, ` + )})` + ); + const dependents = relationDependents(known); + if (dependents.length) { + await tx.execute( + sql`update mirror_pages set source_updated_at = null where mirror_id = ${mirrorId} and entity in (${sql.join( + dependents.map((entity) => sql`${entity}`), + sql`, ` + )})` + ); + } + await tx + .update(m) + .set({ lastSyncedAt: null, mappingGeneration: sql`${m.mappingGeneration} + 1` }) + .where(eq(m.id, mirrorId)); + }); +} + +/** Pause or resume the owner's mirror. Pausing keeps the first `paused_at`. Null when there is no mirror. */ +export async function setMirrorPaused( + ownerUserId: string, + paused: boolean, + options: MirrorDataOptions = {} +): Promise { + const db = await handle(options); + const [row] = await db + .update(m) + .set({ pausedAt: paused ? sql`coalesce(${m.pausedAt}, now())` : null }) + .where(eq(m.ownerUserId, ownerUserId)) + .returning(RECORD_COLUMNS); + return row ? toRecord(row) : null; +} + +// ── The push's own claim (the overlap guard, §3.8 step 1) ─────────── + +/** + * Start a push: set `running_since = now()` if the mirror is connected, not + * paused, its owner may still manage it, and nobody is pushing it (or the last push has been "running" for + * longer than {@link MIRROR_RUN_STALE_MINUTES}, which is a dead one). + * + * The watermark is taken in the same statement, so it is exactly the claim + * time minus {@link MIRROR_WATERMARK_SAFETY_MINUTES}. + */ +export async function claimMirrorRun(mirrorId: string, options: MirrorDataOptions = {}): Promise { + if (!isUuid(mirrorId)) return { skipped: "not_found" }; + const db = await handle(options); + const [row] = await db + .update(m) + .set({ runningSince: sql`now()` }) + .where(and(eq(m.id, mirrorId), ACTIVE, NOT_RUNNING)) + .returning({ + ...RECORD_COLUMNS, + tokenCiphertext: m.tokenCiphertext, + watermark: sql`(now() - ${minutes(MIRROR_WATERMARK_SAFETY_MINUTES)})::text`, + since: sql`${m.lastSyncedAt}::text`, + generation: m.mappingGeneration, + }); + if (row) { + const { tokenCiphertext, watermark, since, generation, ...record } = row; + return { + ...toRecord(record), + tokenCiphertext: tokenCiphertext ?? null, + watermark, + since: since ?? null, + generation: Number(generation), + }; + } + + const [state] = await db + .select({ + connected: sql`(${m.tokenCiphertext} is not null)`, + paused: sql`(${m.pausedAt} is not null)`, + ownerAllowed: sql`${OWNER_ALLOWED}`, + }) + .from(m) + .where(eq(m.id, mirrorId)) + .limit(1); + if (!state) return { skipped: "not_found" }; + if (!bool(state.connected)) return { skipped: "not_connected" }; + if (bool(state.paused)) return { skipped: "paused" }; + if (!bool(state.ownerAllowed)) return { skipped: "owner_not_allowed" }; + return { skipped: "running" }; +} + +/** + * End a push, whatever happened: clear `running_since`, stamp `last_run_at`, + * record the status and error, and — only when `advanceTo` is given and the + * mirror is still at the claim's `generation` — move `last_synced_at` to it. + * A mapping change or reset during the push set `last_synced_at` to null to + * force a full push; overwriting it with this push's watermark would strand + * every row older than it in the newly mapped databases. A partial or failed push passes null, so the rows + * that failed are selected again next time (§3.8 step 5). `pause` pauses the + * mirror (a revoked token, §5.8) and keeps an existing `paused_at`. + * + * Answers `current: false` when the mapping moved on during the push, so the + * caller knows its work did not count as a complete push. + */ +export async function finishMirrorRun( + mirrorId: string, + input: FinishMirrorRunInput, + options: MirrorDataOptions = {} +): Promise<{ current: boolean }> { + if (!isUuid(mirrorId)) return { current: false }; + const db = await handle(options); + const rows = await db + .update(m) + .set({ + runningSince: null, + lastRunAt: sql`now()`, + lastStatus: input.status, + lastError: input.error, + ...(input.advanceTo !== null + ? { + lastSyncedAt: sql`case when ${m.mappingGeneration} = ${input.generation} then ${input.advanceTo}::timestamptz else ${m.lastSyncedAt} end`, + } + : {}), + ...(input.pause ? { pausedAt: sql`coalesce(${m.pausedAt}, now())` } : {}), + }) + .where(eq(m.id, mirrorId)) + .returning({ generation: m.mappingGeneration }); + return { current: rows.length > 0 && Number(rows[0].generation) === input.generation }; +} + +/** + * Free the overlap guard of a claimed run that had nothing to do — no entity + * mapped — without writing a status. `finishMirrorRun` always records one, and + * a mirror that pushed nothing should keep the result it last earned rather + * than show a made-up "ok" or "failed". Stamps `last_run_at`, so a Sync now + * claim is seen as settled. + */ +export async function releaseMirrorRun(mirrorId: string, options: MirrorDataOptions = {}): Promise { + if (!isUuid(mirrorId)) return; + const db = await handle(options); + await db.update(m).set({ runningSince: null, lastRunAt: sql`now()` }).where(eq(m.id, mirrorId)); +} + +// ── Sync now (§8: one push per mirror per 15 minutes) ─────────────── + +/** + * Claim the owner's Sync now: set `sync_requested_at = now()` if the mirror is + * connected, not paused, mapped, not being pushed right now, and was not + * synced on request in the last {@link MIRROR_SYNC_NOW_MINUTES}. A refusal + * says why, and `too_soon` says how many seconds are left. + * + * **A running push refuses the claim** (`running`) rather than spend the + * owner's fifteen minutes on a push whose own claim would be skipped as + * `running` — the page would then show the other push's result as if it + * answered the press. The owner presses again once it finishes. + * + * The owner's role is not re-checked here: the server action has just checked + * `mirror.manage` on the live identity, which is the authority. + */ +export async function claimManualSync( + ownerUserId: string, + options: MirrorDataOptions = {} +): Promise { + const db = await handle(options); + const [claimed] = await db + .update(m) + .set({ syncRequestedAt: sql`now()` }) + .where(and(eq(m.ownerUserId, ownerUserId), CONNECTED, MAPPED, SYNC_WINDOW_OPEN, NOT_RUNNING)) + .returning({ id: m.id }); + if (claimed) return { ok: true, mirrorId: claimed.id }; + + const [state] = await db + .select({ + connected: sql`(${m.tokenCiphertext} is not null)`, + paused: sql`(${m.pausedAt} is not null)`, + mapped: sql`${MAPPED}`, + windowOpen: sql`${SYNC_WINDOW_OPEN}`, + retryAfterSeconds: sql`case when ${SYNC_WINDOW_OPEN} then null else greatest(1, ceil(extract(epoch from (${m.syncRequestedAt} + ${minutes( + MIRROR_SYNC_NOW_MINUTES + )} - now()))))::int end`, + }) + .from(m) + .where(eq(m.ownerUserId, ownerUserId)) + .limit(1); + if (!state) return { ok: false, reason: "not_found" }; + if (!bool(state.connected)) return { ok: false, reason: "not_connected" }; + if (bool(state.paused)) return { ok: false, reason: "paused" }; + if (!bool(state.mapped)) return { ok: false, reason: "not_mapped" }; + if (bool(state.windowOpen)) return { ok: false, reason: "running" }; + const retryAfterSeconds = Number(state.retryAfterSeconds ?? 1); + return { ok: false, reason: "too_soon", retryAfterSeconds: Number.isFinite(retryAfterSeconds) ? retryAfterSeconds : 1 }; +} + +/** Give a Sync now claim back — the workflow could not be started, so it did not count. */ +export async function releaseManualSync(mirrorId: string, options: MirrorDataOptions = {}): Promise { + if (!isUuid(mirrorId)) return; + const db = await handle(options); + await db.update(m).set({ syncRequestedAt: null }).where(eq(m.id, mirrorId)); +} + +// ── Coalescing (§3.8 trigger 1) ───────────────────────────────────── + +/** + * A change happened: claim a coalesced push for every active mirror that does + * not already have one waiting (or whose claim is older than + * {@link MIRROR_COALESCE_STALE_MINUTES}, a run that never finished). Returns + * the ids claimed; an empty list means a push is already on its way, or there + * is nothing to push to, and the caller starts nothing. + */ +export async function claimCoalescedPush(options: MirrorDataOptions = {}): Promise { + const db = await handle(options); + const rows = await db + .update(m) + .set({ pushRequestedAt: sql`now()` }) + .where( + and( + ACTIVE, + sql`(${m.pushRequestedAt} is null or ${m.pushRequestedAt} < now() - ${minutes(MIRROR_COALESCE_STALE_MINUTES)})` + ) + ) + .returning({ id: m.id }); + return rows.map((row) => row.id); +} + +/** Give coalescing claims back — the workflow could not be started. */ +export async function releaseCoalescedPush(ids: string[], options: MirrorDataOptions = {}): Promise { + const valid = ids.filter(isUuid); + if (valid.length === 0) return; + const db = await handle(options); + await db.update(m).set({ pushRequestedAt: null }).where(inArray(m.id, valid)); +} + +/** + * The coalescing run woke up: clear every waiting claim and return the ids of + * the mirrors still active, which it then pushes. Clearing first means a + * change made while those pushes run claims — and schedules — the next one. + */ +export async function takeCoalescedPush(options: MirrorDataOptions = {}): Promise { + const db = await handle(options); + const rows = await db + .update(m) + .set({ pushRequestedAt: null }) + .where(sql`${m.pushRequestedAt} is not null`) + .returning({ id: m.id, active: sql`${ACTIVE}` }); + return rows.filter((row) => bool(row.active)).map((row) => row.id); +} + +// ── The daily backstop (§3.9) ─────────────────────────────────────── + +/** + * Mirrors the daily cron should push: active, not running, and either never + * synced, not `ok` last time, or behind a source table that changed since. + * One `EXISTS` per mirrored table, each a scan of `updated_at` bounded by the + * first match. + */ +export async function listMirrorsDueForBackstop(options: MirrorDataOptions = {}): Promise { + const db = await handle(options); + const changed = (table: string) => + sql`exists (select 1 from ${sql.raw(`"${table}"`)} as s where s.updated_at > ${m.lastSyncedAt})`; + const rows = await db + .select({ id: m.id }) + .from(m) + .where( + and( + ACTIVE, + NOT_RUNNING, + sql`(${m.lastSyncedAt} is null + or ${m.lastStatus} is distinct from 'ok' + or ${changed("categories")} + or ${changed("locations")} + or ${changed("tools")} + or ${changed("units")} + or ${changed("resources")} + or ${changed("maintenance_logs")} + or ${changed("projects")})` + ) + ); + return rows.map((row) => row.id); +} diff --git a/v5/src/lib/data/pending-tools.ts b/v5/src/lib/data/pending-tools.ts index 67ad924..41072ce 100644 --- a/v5/src/lib/data/pending-tools.ts +++ b/v5/src/lib/data/pending-tools.ts @@ -197,6 +197,8 @@ export type ApprovePendingResult = slug: string; unitId: string | null; resourcesCreated: number; + /** The resources created, for the manual archive to copy after the commit. */ + resourceIds: string[]; photosMoved: number; published: boolean; /** True when a low-confidence grade was overridden with a note. */ @@ -1045,6 +1047,7 @@ export async function approvePendingTool( slug: created.slug, unitId, resourcesCreated: created.resourceIds.length, + resourceIds: created.resourceIds, photosMoved, published: input.publish, overridden, diff --git a/v5/src/lib/data/resources.test.ts b/v5/src/lib/data/resources.test.ts index 8702584..6567b57 100644 --- a/v5/src/lib/data/resources.test.ts +++ b/v5/src/lib/data/resources.test.ts @@ -1,5 +1,6 @@ // @vitest-environment node import { createPgliteDb } from "../db/pglite"; +import { manualSourceKey } from "./manual-archives"; import { attachments, resources, tools } from "../db/schema/index"; import type { Db } from "../db/types"; import { @@ -81,6 +82,7 @@ describe("listResourcesForTool", () => { url: "https://x.test/sop.pdf", notes: "Read before the first print.", fileUrls: [], + archivedUrl: null, }, ]); }); @@ -97,6 +99,18 @@ describe("listResourcesForTool", () => { ]); }); + it("sets the archived copy of the current link apart, and drops a stale one", async () => { + const url = "https://maker.test/manual.pdf"; + const id = await insertResource({ title: "Manual", type: "Manual", url }); + await attachFile(id, { publicUrl: "https://blob.test/archive.pdf", sourceKey: manualSourceKey(id, url) }); + await attachFile(id, { publicUrl: "https://blob.test/stale.pdf", sourceKey: manualSourceKey(id, "https://maker.test/old.pdf") }); + await attachFile(id, { publicUrl: "https://blob.test/upload.pdf", position: 1 }); + + const [resource] = await listResourcesForTool(form4, { db }); + expect(resource.archivedUrl).toBe("https://blob.test/archive.pdf"); + expect(resource.fileUrls).toEqual(["https://blob.test/upload.pdf"]); + }); + it("leaves out private files, which no visitor could open either", async () => { const id = await insertResource({ title: "Incident photos", url: null }); await attachFile(id, { access: "private", publicUrl: null }); diff --git a/v5/src/lib/data/resources.ts b/v5/src/lib/data/resources.ts index 2189c4f..156ee33 100644 --- a/v5/src/lib/data/resources.ts +++ b/v5/src/lib/data/resources.ts @@ -3,6 +3,7 @@ import { getDb } from "../db/client.ts"; import { attachments, resources } from "../db/schema/index.ts"; import type { Db } from "../db/types.ts"; import { claimAttachments, releaseAttachments } from "./attachments.ts"; +import { isManualArchiveKey, manualSourceKey } from "./manual-archives.ts"; import { isUuid } from "./uuid.ts"; import type { Refused } from "./write-result.ts"; @@ -33,8 +34,14 @@ export interface ToolResource { type: string | null; url: string | null; notes: string | null; - /** Public URLs of the resource's attachments, cover first. */ + /** Public URLs of the resource's attachments, cover first — archived manuals excluded. */ fileUrls: string[]; + /** + * The public archived copy of the PDF `url` points at, when the manual + * archive has made one (`./manual-archives.ts`). Prefer it: it outlives + * the manufacturer's link. + */ + archivedUrl: string | null; } export interface ResourceQueryOptions { @@ -87,12 +94,9 @@ async function loadResources(db: Db, where: SQL | undefined): Promise row.id) - ); + const filesByResource = await loadFileUrls(db, rows); - return rows.map((row) => ({ ...row, fileUrls: filesByResource.get(row.id) ?? [] })); + return rows.map((row) => ({ ...row, ...(filesByResource.get(row.id) ?? NO_FILES) })); } /** One resource as the editor lists it — unpublished ones included. */ @@ -103,8 +107,10 @@ export interface EditorResource { url: string | null; notes: string | null; published: boolean; - /** Public URLs of the files hanging off it — a manual is usually one PDF. */ + /** Public URLs of the files hanging off it — a manual is usually one PDF. Archived manuals excluded. */ fileUrls: string[]; + /** The archived copy of the PDF `url` points at, if the manual archive has made one. */ + archivedUrl?: string | null; } /** @@ -136,12 +142,9 @@ export async function listResourcesForEditor( if (rows.length === 0) return []; - const filesByResource = await loadFileUrls( - db, - rows.map((row) => row.id) - ); + const filesByResource = await loadFileUrls(db, rows); - return rows.map((row) => ({ ...row, fileUrls: filesByResource.get(row.id) ?? [] })); + return rows.map((row) => ({ ...row, ...(filesByResource.get(row.id) ?? NO_FILES) })); } // ── Writes (spec §4.6, §5.3(3)) ───────────────────────────────────── @@ -314,31 +317,59 @@ function emptyToNull(value: string | null): string | null { return trimmed || null; } +/** A resource's public files, as the two reads above carry them. */ +interface ResourceFiles { + fileUrls: string[]; + archivedUrl: string | null; +} + +const NO_FILES: ResourceFiles = { fileUrls: [], archivedUrl: null }; + /** * Public attachment URLs owned by these resources, grouped by resource id and * ordered by `position`. A private file has no URL a visitor could open, and * the model is given nothing a visitor could not read. + * + * An archived manual is set apart as `archivedUrl` — only the copy of the link + * the resource carries now; a stale copy of an edited link is dropped — so a + * caller never counts one manual twice. */ -async function loadFileUrls(db: Db, resourceIds: string[]): Promise> { +async function loadFileUrls( + db: Db, + owners: ReadonlyArray<{ id: string; url: string | null }> +): Promise> { const rows = await db - .select({ ownerId: attachments.ownerId, publicUrl: attachments.publicUrl }) + .select({ ownerId: attachments.ownerId, publicUrl: attachments.publicUrl, sourceKey: attachments.sourceKey }) .from(attachments) .where( and( eq(attachments.ownerType, "resource"), - inArray(attachments.ownerId, resourceIds), + inArray( + attachments.ownerId, + owners.map((owner) => owner.id) + ), eq(attachments.access, "public"), isNotNull(attachments.publicUrl) ) ) .orderBy(asc(attachments.position), asc(attachments.id)); - const map = new Map(); + const currentKey = new Map( + owners.map((owner) => [owner.id, owner.url ? manualSourceKey(owner.id, owner.url) : null]) + ); + const map = new Map(); for (const row of rows) { if (!row.ownerId || !row.publicUrl) continue; - const list = map.get(row.ownerId); - if (list) list.push(row.publicUrl); - else map.set(row.ownerId, [row.publicUrl]); + let entry = map.get(row.ownerId); + if (!entry) { + entry = { fileUrls: [], archivedUrl: null }; + map.set(row.ownerId, entry); + } + if (isManualArchiveKey(row.sourceKey)) { + if (row.sourceKey === currentKey.get(row.ownerId)) entry.archivedUrl ??= row.publicUrl; + continue; + } + entry.fileUrls.push(row.publicUrl); } return map; } diff --git a/v5/src/lib/data/users.ts b/v5/src/lib/data/users.ts index 19be66b..190438a 100644 --- a/v5/src/lib/data/users.ts +++ b/v5/src/lib/data/users.ts @@ -21,7 +21,10 @@ import type { Db } from "../db/types.ts"; * Emails *are* returned, unlike everywhere else in the app: this is the one * surface whose whole job is telling a super admin which account is which, and * a roster of display names cannot do that. It goes no further — not into a - * prompt, not into the mirror, not into a log line (spec §8). + * prompt and not into a log line (spec §8). This roster is not the mirror's + * source: the Notion mirror selects the emails it carries (maintenance reporter + * and assignee, project author) itself, in `mirror/source.ts`, as the + * 2026-09-23 amendment decided. * * Relative imports with `.ts` extensions, no `@/` alias and no `"server-only"`, * like every other module under `src/lib/data/`. diff --git a/v5/src/lib/db/migrations/0007_notion_mirror.sql b/v5/src/lib/db/migrations/0007_notion_mirror.sql new file mode 100644 index 0000000..1f766fc --- /dev/null +++ b/v5/src/lib/db/migrations/0007_notion_mirror.sql @@ -0,0 +1,38 @@ +CREATE TABLE "mirror_pages" ( + "mirror_id" uuid NOT NULL, + "entity" text NOT NULL, + "entity_id" uuid NOT NULL, + "notion_page_id" text NOT NULL, + "pushed_at" timestamp with time zone DEFAULT now() NOT NULL, + "source_updated_at" timestamp with time zone, + CONSTRAINT "mirror_pages_mirror_id_entity_entity_id_pk" PRIMARY KEY("mirror_id","entity","entity_id"), + CONSTRAINT "mirror_pages_entity_check" CHECK ("entity" in ('categories', 'locations', 'tools', 'units', 'resources', 'maintenance', 'projects')) +); +--> statement-breakpoint +CREATE TABLE "notion_mirrors" ( + "id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL, + "owner_user_id" text NOT NULL, + "token_ciphertext" "bytea", + "parent_page_id" text NOT NULL, + "parent_page_title" text, + "mapping" jsonb DEFAULT '{}'::jsonb NOT NULL, + "mapping_generation" integer DEFAULT 0 NOT NULL, + "paused_at" timestamp with time zone, + "running_since" timestamp with time zone, + "push_requested_at" timestamp with time zone, + "sync_requested_at" timestamp with time zone, + "last_synced_at" timestamp with time zone, + "last_run_at" timestamp with time zone, + "last_status" text, + "last_error" jsonb, + "created_at" timestamp with time zone DEFAULT now() NOT NULL, + "updated_at" timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT "notion_mirrors_owner_user_id_unique" UNIQUE("owner_user_id"), + CONSTRAINT "notion_mirrors_last_status_check" CHECK ("last_status" in ('ok', 'partial', 'failed')) +); +--> statement-breakpoint +ALTER TABLE "mirror_pages" ADD CONSTRAINT "mirror_pages_mirror_id_notion_mirrors_id_fk" FOREIGN KEY ("mirror_id") REFERENCES "public"."notion_mirrors"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint +ALTER TABLE "notion_mirrors" ADD CONSTRAINT "notion_mirrors_owner_user_id_user_id_fk" FOREIGN KEY ("owner_user_id") REFERENCES "public"."user"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint +-- Hand-appended, as in 0002: updated_at is maintained by the database, not the +-- ORM (spec §4). drizzle-kit does not generate triggers. +CREATE TRIGGER notion_mirrors_set_updated_at BEFORE UPDATE ON "notion_mirrors" FOR EACH ROW EXECUTE FUNCTION set_updated_at(); diff --git a/v5/src/lib/db/migrations/meta/0007_snapshot.json b/v5/src/lib/db/migrations/meta/0007_snapshot.json new file mode 100644 index 0000000..bc4d041 --- /dev/null +++ b/v5/src/lib/db/migrations/meta/0007_snapshot.json @@ -0,0 +1,2972 @@ +{ + "id": "613cfc56-595d-401e-9259-b08ca35eaf4e", + "prevId": "980c498e-3e68-47f2-9c90-7ddb10548f98", + "version": "7", + "dialect": "postgresql", + "tables": { + "public.account": { + "name": "account", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "account_id": { + "name": "account_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "provider_id": { + "name": "provider_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "access_token": { + "name": "access_token", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "refresh_token": { + "name": "refresh_token", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "id_token": { + "name": "id_token", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "access_token_expires_at": { + "name": "access_token_expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "refresh_token_expires_at": { + "name": "refresh_token_expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "scope": { + "name": "scope", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "password": { + "name": "password", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "account_user_idx": { + "name": "account_user_idx", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "account_user_id_user_id_fk": { + "name": "account_user_id_user_id_fk", + "tableFrom": "account", + "tableTo": "user", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.session": { + "name": "session", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "token": { + "name": "token", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "ip_address": { + "name": "ip_address", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "user_agent": { + "name": "user_agent", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "impersonated_by": { + "name": "impersonated_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "session_user_idx": { + "name": "session_user_idx", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "session_user_id_user_id_fk": { + "name": "session_user_id_user_id_fk", + "tableFrom": "session", + "tableTo": "user", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "session_token_unique": { + "name": "session_token_unique", + "nullsNotDistinct": false, + "columns": [ + "token" + ] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.user": { + "name": "user", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "email": { + "name": "email", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "email_verified": { + "name": "email_verified", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "image": { + "name": "image", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": false, + "default": "'user'" + }, + "banned": { + "name": "banned", + "type": "boolean", + "primaryKey": false, + "notNull": false, + "default": false + }, + "ban_reason": { + "name": "ban_reason", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "ban_expires": { + "name": "ban_expires", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "user_email_idx": { + "name": "user_email_idx", + "columns": [ + { + "expression": "email", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "user_email_unique": { + "name": "user_email_unique", + "nullsNotDistinct": false, + "columns": [ + "email" + ] + } + }, + "policies": {}, + "checkConstraints": { + "user_role_check": { + "name": "user_role_check", + "value": "\"role\" in ('user', 'admin', 'super_admin')" + } + }, + "isRLSEnabled": false + }, + "public.verification": { + "name": "verification", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "identifier": { + "name": "identifier", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "value": { + "name": "value", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "verification_identifier_idx": { + "name": "verification_identifier_idx", + "columns": [ + { + "expression": "identifier", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.categories": { + "name": "categories", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "group": { + "name": "group", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "notion_page_id": { + "name": "notion_page_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "updated_by": { + "name": "updated_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "categories_name_group_key": { + "name": "categories_name_group_key", + "columns": [ + { + "expression": "lower(\"name\")", + "asc": true, + "isExpression": true, + "nulls": "last" + }, + { + "expression": "lower(coalesce(\"group\", ''))", + "asc": true, + "isExpression": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "categories_created_by_user_id_fk": { + "name": "categories_created_by_user_id_fk", + "tableFrom": "categories", + "tableTo": "user", + "columnsFrom": [ + "created_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "categories_updated_by_user_id_fk": { + "name": "categories_updated_by_user_id_fk", + "tableFrom": "categories", + "tableTo": "user", + "columnsFrom": [ + "updated_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "categories_notion_page_id_unique": { + "name": "categories_notion_page_id_unique", + "nullsNotDistinct": false, + "columns": [ + "notion_page_id" + ] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.locations": { + "name": "locations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "room": { + "name": "room", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "zone": { + "name": "zone", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "map_tag": { + "name": "map_tag", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "notion_page_id": { + "name": "notion_page_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "updated_by": { + "name": "updated_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "locations_room_zone_key": { + "name": "locations_room_zone_key", + "columns": [ + { + "expression": "lower(\"room\")", + "asc": true, + "isExpression": true, + "nulls": "last" + }, + { + "expression": "lower(\"zone\")", + "asc": true, + "isExpression": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "locations_created_by_user_id_fk": { + "name": "locations_created_by_user_id_fk", + "tableFrom": "locations", + "tableTo": "user", + "columnsFrom": [ + "created_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "locations_updated_by_user_id_fk": { + "name": "locations_updated_by_user_id_fk", + "tableFrom": "locations", + "tableTo": "user", + "columnsFrom": [ + "updated_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "locations_map_tag_unique": { + "name": "locations_map_tag_unique", + "nullsNotDistinct": false, + "columns": [ + "map_tag" + ] + }, + "locations_notion_page_id_unique": { + "name": "locations_notion_page_id_unique", + "nullsNotDistinct": false, + "columns": [ + "notion_page_id" + ] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.tools": { + "name": "tools", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "slug": { + "name": "slug", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "category_id": { + "name": "category_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "location_id": { + "name": "location_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "materials": { + "name": "materials", + "type": "text[]", + "primaryKey": false, + "notNull": true, + "default": "'{}'::text[]" + }, + "ppe_required": { + "name": "ppe_required", + "type": "text[]", + "primaryKey": false, + "notNull": true, + "default": "'{}'::text[]" + }, + "tags": { + "name": "tags", + "type": "text[]", + "primaryKey": false, + "notNull": true, + "default": "'{}'::text[]" + }, + "training_required": { + "name": "training_required", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "use_restrictions": { + "name": "use_restrictions", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "emergency_stop": { + "name": "emergency_stop", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "notes": { + "name": "notes", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "published": { + "name": "published", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "archived_at": { + "name": "archived_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "last_reviewed_at": { + "name": "last_reviewed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "last_reviewed_by": { + "name": "last_reviewed_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "notion_page_id": { + "name": "notion_page_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "updated_by": { + "name": "updated_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "tools_name_trgm_idx": { + "name": "tools_name_trgm_idx", + "columns": [ + { + "expression": "\"name\" gin_trgm_ops", + "asc": true, + "isExpression": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "gin", + "with": {} + }, + "tools_category_idx": { + "name": "tools_category_idx", + "columns": [ + { + "expression": "category_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "tools_location_idx": { + "name": "tools_location_idx", + "columns": [ + { + "expression": "location_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "tools_category_id_categories_id_fk": { + "name": "tools_category_id_categories_id_fk", + "tableFrom": "tools", + "tableTo": "categories", + "columnsFrom": [ + "category_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "tools_location_id_locations_id_fk": { + "name": "tools_location_id_locations_id_fk", + "tableFrom": "tools", + "tableTo": "locations", + "columnsFrom": [ + "location_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "tools_last_reviewed_by_user_id_fk": { + "name": "tools_last_reviewed_by_user_id_fk", + "tableFrom": "tools", + "tableTo": "user", + "columnsFrom": [ + "last_reviewed_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "tools_created_by_user_id_fk": { + "name": "tools_created_by_user_id_fk", + "tableFrom": "tools", + "tableTo": "user", + "columnsFrom": [ + "created_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "tools_updated_by_user_id_fk": { + "name": "tools_updated_by_user_id_fk", + "tableFrom": "tools", + "tableTo": "user", + "columnsFrom": [ + "updated_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "tools_slug_unique": { + "name": "tools_slug_unique", + "nullsNotDistinct": false, + "columns": [ + "slug" + ] + }, + "tools_notion_page_id_unique": { + "name": "tools_notion_page_id_unique", + "nullsNotDistinct": false, + "columns": [ + "notion_page_id" + ] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.units": { + "name": "units", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "tool_id": { + "name": "tool_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "unit_label": { + "name": "unit_label", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "serial_number": { + "name": "serial_number", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "asset_tag": { + "name": "asset_tag", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'available'" + }, + "condition": { + "name": "condition", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "date_acquired": { + "name": "date_acquired", + "type": "date", + "primaryKey": false, + "notNull": false + }, + "notes": { + "name": "notes", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "notion_page_id": { + "name": "notion_page_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "updated_by": { + "name": "updated_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "units_tool_idx": { + "name": "units_tool_idx", + "columns": [ + { + "expression": "tool_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "units_tool_serial_key": { + "name": "units_tool_serial_key", + "columns": [ + { + "expression": "tool_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "lower(\"serial_number\")", + "asc": true, + "isExpression": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"units\".\"serial_number\" is not null", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "units_tool_id_tools_id_fk": { + "name": "units_tool_id_tools_id_fk", + "tableFrom": "units", + "tableTo": "tools", + "columnsFrom": [ + "tool_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "units_created_by_user_id_fk": { + "name": "units_created_by_user_id_fk", + "tableFrom": "units", + "tableTo": "user", + "columnsFrom": [ + "created_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "units_updated_by_user_id_fk": { + "name": "units_updated_by_user_id_fk", + "tableFrom": "units", + "tableTo": "user", + "columnsFrom": [ + "updated_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "units_notion_page_id_unique": { + "name": "units_notion_page_id_unique", + "nullsNotDistinct": false, + "columns": [ + "notion_page_id" + ] + } + }, + "policies": {}, + "checkConstraints": { + "units_status_check": { + "name": "units_status_check", + "value": "\"status\" in ('available', 'in_use', 'under_maintenance', 'out_of_service', 'retired')" + }, + "units_condition_check": { + "name": "units_condition_check", + "value": "\"condition\" in ('excellent', 'good', 'fair', 'needs_repair', 'new')" + } + }, + "isRLSEnabled": false + }, + "public.resources": { + "name": "resources", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "tool_id": { + "name": "tool_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "url": { + "name": "url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "notes": { + "name": "notes", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "published": { + "name": "published", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": true + }, + "notion_page_id": { + "name": "notion_page_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "updated_by": { + "name": "updated_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "resources_tool_idx": { + "name": "resources_tool_idx", + "columns": [ + { + "expression": "tool_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "resources_tool_id_tools_id_fk": { + "name": "resources_tool_id_tools_id_fk", + "tableFrom": "resources", + "tableTo": "tools", + "columnsFrom": [ + "tool_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "resources_created_by_user_id_fk": { + "name": "resources_created_by_user_id_fk", + "tableFrom": "resources", + "tableTo": "user", + "columnsFrom": [ + "created_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "resources_updated_by_user_id_fk": { + "name": "resources_updated_by_user_id_fk", + "tableFrom": "resources", + "tableTo": "user", + "columnsFrom": [ + "updated_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "resources_notion_page_id_unique": { + "name": "resources_notion_page_id_unique", + "nullsNotDistinct": false, + "columns": [ + "notion_page_id" + ] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.attachments": { + "name": "attachments", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "owner_type": { + "name": "owner_type", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "owner_id": { + "name": "owner_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "position": { + "name": "position", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "blob_pathname": { + "name": "blob_pathname", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "access": { + "name": "access", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "public_url": { + "name": "public_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "content_type": { + "name": "content_type", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "size_bytes": { + "name": "size_bytes", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "width": { + "name": "width", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "height": { + "name": "height", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "original_filename": { + "name": "original_filename", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "source_key": { + "name": "source_key", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "uploaded_by": { + "name": "uploaded_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "attachments_owner_idx": { + "name": "attachments_owner_idx", + "columns": [ + { + "expression": "owner_type", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "owner_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "position", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "attachments_source_key_unique": { + "name": "attachments_source_key_unique", + "nullsNotDistinct": false, + "columns": [ + "source_key" + ] + } + }, + "policies": {}, + "checkConstraints": { + "attachments_owner_type_check": { + "name": "attachments_owner_type_check", + "value": "\"owner_type\" in ('tool', 'resource', 'maintenance_log', 'project', 'pending_tool')" + }, + "attachments_access_check": { + "name": "attachments_access_check", + "value": "\"access\" in ('public', 'private')" + } + }, + "isRLSEnabled": false + }, + "public.maintenance_logs": { + "name": "maintenance_logs", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "type": { + "name": "type", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "priority": { + "name": "priority", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'open'" + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "resolution": { + "name": "resolution", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "unit_id": { + "name": "unit_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "tool_id": { + "name": "tool_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "tool_name": { + "name": "tool_name", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "unit_label": { + "name": "unit_label", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "reported_by_name": { + "name": "reported_by_name", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "reported_by_email": { + "name": "reported_by_email", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "reported_by_user_id": { + "name": "reported_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "assigned_to_user_id": { + "name": "assigned_to_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "assigned_to_name": { + "name": "assigned_to_name", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "date_reported": { + "name": "date_reported", + "type": "date", + "primaryKey": false, + "notNull": false + }, + "date_resolved": { + "name": "date_resolved", + "type": "date", + "primaryKey": false, + "notNull": false + }, + "notion_page_id": { + "name": "notion_page_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "updated_by": { + "name": "updated_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "maintenance_logs_unit_idx": { + "name": "maintenance_logs_unit_idx", + "columns": [ + { + "expression": "unit_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "maintenance_logs_tool_idx": { + "name": "maintenance_logs_tool_idx", + "columns": [ + { + "expression": "tool_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "maintenance_logs_status_idx": { + "name": "maintenance_logs_status_idx", + "columns": [ + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "maintenance_logs_unit_id_units_id_fk": { + "name": "maintenance_logs_unit_id_units_id_fk", + "tableFrom": "maintenance_logs", + "tableTo": "units", + "columnsFrom": [ + "unit_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "maintenance_logs_tool_id_tools_id_fk": { + "name": "maintenance_logs_tool_id_tools_id_fk", + "tableFrom": "maintenance_logs", + "tableTo": "tools", + "columnsFrom": [ + "tool_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "maintenance_logs_assigned_to_user_id_user_id_fk": { + "name": "maintenance_logs_assigned_to_user_id_user_id_fk", + "tableFrom": "maintenance_logs", + "tableTo": "user", + "columnsFrom": [ + "assigned_to_user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "maintenance_logs_created_by_user_id_fk": { + "name": "maintenance_logs_created_by_user_id_fk", + "tableFrom": "maintenance_logs", + "tableTo": "user", + "columnsFrom": [ + "created_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "maintenance_logs_updated_by_user_id_fk": { + "name": "maintenance_logs_updated_by_user_id_fk", + "tableFrom": "maintenance_logs", + "tableTo": "user", + "columnsFrom": [ + "updated_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "maintenance_logs_notion_page_id_unique": { + "name": "maintenance_logs_notion_page_id_unique", + "nullsNotDistinct": false, + "columns": [ + "notion_page_id" + ] + } + }, + "policies": {}, + "checkConstraints": { + "maintenance_logs_type_check": { + "name": "maintenance_logs_type_check", + "value": "\"type\" in ('issue_report', 'preventive_maintenance', 'repair', 'inspection', 'calibration')" + }, + "maintenance_logs_priority_check": { + "name": "maintenance_logs_priority_check", + "value": "\"priority\" in ('low', 'medium', 'high', 'critical')" + }, + "maintenance_logs_status_check": { + "name": "maintenance_logs_status_check", + "value": "\"status\" in ('open', 'in_progress', 'resolved', 'closed')" + } + }, + "isRLSEnabled": false + }, + "public.feedback": { + "name": "feedback", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "tool_id": { + "name": "tool_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "field_flagged": { + "name": "field_flagged", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "issue_description": { + "name": "issue_description", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "suggested_fix": { + "name": "suggested_fix", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "reporter_name": { + "name": "reporter_name", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "reporter_email": { + "name": "reporter_email", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "reporter_user_id": { + "name": "reporter_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'new'" + }, + "notion_page_id": { + "name": "notion_page_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "updated_by": { + "name": "updated_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "feedback_tool_idx": { + "name": "feedback_tool_idx", + "columns": [ + { + "expression": "tool_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "feedback_status_idx": { + "name": "feedback_status_idx", + "columns": [ + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "feedback_tool_id_tools_id_fk": { + "name": "feedback_tool_id_tools_id_fk", + "tableFrom": "feedback", + "tableTo": "tools", + "columnsFrom": [ + "tool_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "feedback_created_by_user_id_fk": { + "name": "feedback_created_by_user_id_fk", + "tableFrom": "feedback", + "tableTo": "user", + "columnsFrom": [ + "created_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "feedback_updated_by_user_id_fk": { + "name": "feedback_updated_by_user_id_fk", + "tableFrom": "feedback", + "tableTo": "user", + "columnsFrom": [ + "updated_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "feedback_notion_page_id_unique": { + "name": "feedback_notion_page_id_unique", + "nullsNotDistinct": false, + "columns": [ + "notion_page_id" + ] + } + }, + "policies": {}, + "checkConstraints": { + "feedback_field_flagged_check": { + "name": "feedback_field_flagged_check", + "value": "\"field_flagged\" in ('description', 'image', 'name', 'category', 'location', 'materials', 'safety_info')" + }, + "feedback_status_check": { + "name": "feedback_status_check", + "value": "\"status\" in ('new', 'reviewed', 'fixed', 'dismissed')" + } + }, + "isRLSEnabled": false + }, + "public.project_tools": { + "name": "project_tools", + "schema": "", + "columns": { + "project_id": { + "name": "project_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "tool_id": { + "name": "tool_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + } + }, + "indexes": { + "project_tools_tool_idx": { + "name": "project_tools_tool_idx", + "columns": [ + { + "expression": "tool_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "project_tools_project_id_projects_id_fk": { + "name": "project_tools_project_id_projects_id_fk", + "tableFrom": "project_tools", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "project_tools_tool_id_tools_id_fk": { + "name": "project_tools_tool_id_tools_id_fk", + "tableFrom": "project_tools", + "tableTo": "tools", + "columnsFrom": [ + "tool_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "project_tools_project_id_tool_id_pk": { + "name": "project_tools_project_id_tool_id_pk", + "columns": [ + "project_id", + "tool_id" + ] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.projects": { + "name": "projects", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "slug": { + "name": "slug", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "body": { + "name": "body", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "''" + }, + "link": { + "name": "link", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "materials": { + "name": "materials", + "type": "text[]", + "primaryKey": false, + "notNull": true, + "default": "'{}'::text[]" + }, + "author_user_id": { + "name": "author_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "author_name": { + "name": "author_name", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "published": { + "name": "published", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "published_at": { + "name": "published_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "published_by": { + "name": "published_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "notion_page_id": { + "name": "notion_page_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "updated_by": { + "name": "updated_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "projects_published_idx": { + "name": "projects_published_idx", + "columns": [ + { + "expression": "published", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "projects_published_by_user_id_fk": { + "name": "projects_published_by_user_id_fk", + "tableFrom": "projects", + "tableTo": "user", + "columnsFrom": [ + "published_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "projects_created_by_user_id_fk": { + "name": "projects_created_by_user_id_fk", + "tableFrom": "projects", + "tableTo": "user", + "columnsFrom": [ + "created_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "projects_updated_by_user_id_fk": { + "name": "projects_updated_by_user_id_fk", + "tableFrom": "projects", + "tableTo": "user", + "columnsFrom": [ + "updated_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "projects_slug_unique": { + "name": "projects_slug_unique", + "nullsNotDistinct": false, + "columns": [ + "slug" + ] + }, + "projects_notion_page_id_unique": { + "name": "projects_notion_page_id_unique", + "nullsNotDistinct": false, + "columns": [ + "notion_page_id" + ] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.audit_events": { + "name": "audit_events", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "at": { + "name": "at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "actor_user_id": { + "name": "actor_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "action": { + "name": "action", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "subject_type": { + "name": "subject_type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "subject_id": { + "name": "subject_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "detail": { + "name": "detail", + "type": "jsonb", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "audit_events_subject_idx": { + "name": "audit_events_subject_idx", + "columns": [ + { + "expression": "subject_type", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "subject_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "audit_events_at_idx": { + "name": "audit_events_at_idx", + "columns": [ + { + "expression": "at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "audit_events_actor_user_id_user_id_fk": { + "name": "audit_events_actor_user_id_user_id_fk", + "tableFrom": "audit_events", + "tableTo": "user", + "columnsFrom": [ + "actor_user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.pending_tools": { + "name": "pending_tools", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "batch_id": { + "name": "batch_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'identified'" + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "brand": { + "name": "brand", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "category_hint": { + "name": "category_hint", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "location_hint": { + "name": "location_hint", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "serial_number": { + "name": "serial_number", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "duplicate_of_tool_id": { + "name": "duplicate_of_tool_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "duplicate_of_pending_id": { + "name": "duplicate_of_pending_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "duplicate_resolution": { + "name": "duplicate_resolution", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "research": { + "name": "research", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "research_error": { + "name": "research_error", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "workflow_run_id": { + "name": "workflow_run_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "research_request_id": { + "name": "research_request_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "research_requested_by": { + "name": "research_requested_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "research_requested_at": { + "name": "research_requested_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_by": { + "name": "created_by", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "approved_by": { + "name": "approved_by", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "approved_at": { + "name": "approved_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "approval_note": { + "name": "approval_note", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_tool_id": { + "name": "created_tool_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "created_unit_id": { + "name": "created_unit_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "pending_tools_status_idx": { + "name": "pending_tools_status_idx", + "columns": [ + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "pending_tools_batch_idx": { + "name": "pending_tools_batch_idx", + "columns": [ + { + "expression": "batch_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "pending_tools_requested_idx": { + "name": "pending_tools_requested_idx", + "columns": [ + { + "expression": "research_requested_by", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "research_requested_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "pending_tools_name_trgm_idx": { + "name": "pending_tools_name_trgm_idx", + "columns": [ + { + "expression": "\"name\" gin_trgm_ops", + "asc": true, + "isExpression": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "gin", + "with": {} + } + }, + "foreignKeys": { + "pending_tools_duplicate_of_tool_id_tools_id_fk": { + "name": "pending_tools_duplicate_of_tool_id_tools_id_fk", + "tableFrom": "pending_tools", + "tableTo": "tools", + "columnsFrom": [ + "duplicate_of_tool_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "pending_tools_duplicate_of_pending_id_pending_tools_id_fk": { + "name": "pending_tools_duplicate_of_pending_id_pending_tools_id_fk", + "tableFrom": "pending_tools", + "tableTo": "pending_tools", + "columnsFrom": [ + "duplicate_of_pending_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "pending_tools_research_requested_by_user_id_fk": { + "name": "pending_tools_research_requested_by_user_id_fk", + "tableFrom": "pending_tools", + "tableTo": "user", + "columnsFrom": [ + "research_requested_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "pending_tools_created_by_user_id_fk": { + "name": "pending_tools_created_by_user_id_fk", + "tableFrom": "pending_tools", + "tableTo": "user", + "columnsFrom": [ + "created_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "pending_tools_approved_by_user_id_fk": { + "name": "pending_tools_approved_by_user_id_fk", + "tableFrom": "pending_tools", + "tableTo": "user", + "columnsFrom": [ + "approved_by" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "pending_tools_created_tool_id_tools_id_fk": { + "name": "pending_tools_created_tool_id_tools_id_fk", + "tableFrom": "pending_tools", + "tableTo": "tools", + "columnsFrom": [ + "created_tool_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, + "pending_tools_created_unit_id_units_id_fk": { + "name": "pending_tools_created_unit_id_units_id_fk", + "tableFrom": "pending_tools", + "tableTo": "units", + "columnsFrom": [ + "created_unit_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "pending_tools_status_check": { + "name": "pending_tools_status_check", + "value": "\"status\" in ('identified', 'queued', 'researching', 'researched', 'failed', 'approved', 'discarded')" + }, + "pending_tools_duplicate_resolution_check": { + "name": "pending_tools_duplicate_resolution_check", + "value": "\"duplicate_resolution\" in ('new_tool', 'add_unit', 'discard')" + } + }, + "isRLSEnabled": false + }, + "public.research_requests": { + "name": "research_requests", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "request_id": { + "name": "request_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "pending_tool_id": { + "name": "pending_tool_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "requested_at": { + "name": "requested_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "research_requests_user_idx": { + "name": "research_requests_user_idx", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "requested_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "research_requests_user_id_user_id_fk": { + "name": "research_requests_user_id_user_id_fk", + "tableFrom": "research_requests", + "tableTo": "user", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "research_requests_pending_tool_id_pending_tools_id_fk": { + "name": "research_requests_pending_tool_id_pending_tools_id_fk", + "tableFrom": "research_requests", + "tableTo": "pending_tools", + "columnsFrom": [ + "pending_tool_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.mirror_pages": { + "name": "mirror_pages", + "schema": "", + "columns": { + "mirror_id": { + "name": "mirror_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "entity": { + "name": "entity", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "entity_id": { + "name": "entity_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "notion_page_id": { + "name": "notion_page_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "pushed_at": { + "name": "pushed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "source_updated_at": { + "name": "source_updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + } + }, + "indexes": {}, + "foreignKeys": { + "mirror_pages_mirror_id_notion_mirrors_id_fk": { + "name": "mirror_pages_mirror_id_notion_mirrors_id_fk", + "tableFrom": "mirror_pages", + "tableTo": "notion_mirrors", + "columnsFrom": [ + "mirror_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "mirror_pages_mirror_id_entity_entity_id_pk": { + "name": "mirror_pages_mirror_id_entity_entity_id_pk", + "columns": [ + "mirror_id", + "entity", + "entity_id" + ] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "mirror_pages_entity_check": { + "name": "mirror_pages_entity_check", + "value": "\"entity\" in ('categories', 'locations', 'tools', 'units', 'resources', 'maintenance', 'projects')" + } + }, + "isRLSEnabled": false + }, + "public.notion_mirrors": { + "name": "notion_mirrors", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "owner_user_id": { + "name": "owner_user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "token_ciphertext": { + "name": "token_ciphertext", + "type": "bytea", + "primaryKey": false, + "notNull": false + }, + "parent_page_id": { + "name": "parent_page_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "parent_page_title": { + "name": "parent_page_title", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "mapping": { + "name": "mapping", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'{}'::jsonb" + }, + "mapping_generation": { + "name": "mapping_generation", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "paused_at": { + "name": "paused_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "running_since": { + "name": "running_since", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "push_requested_at": { + "name": "push_requested_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "sync_requested_at": { + "name": "sync_requested_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "last_synced_at": { + "name": "last_synced_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "last_run_at": { + "name": "last_run_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "last_status": { + "name": "last_status", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "last_error": { + "name": "last_error", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": { + "notion_mirrors_owner_user_id_user_id_fk": { + "name": "notion_mirrors_owner_user_id_user_id_fk", + "tableFrom": "notion_mirrors", + "tableTo": "user", + "columnsFrom": [ + "owner_user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "notion_mirrors_owner_user_id_unique": { + "name": "notion_mirrors_owner_user_id_unique", + "nullsNotDistinct": false, + "columns": [ + "owner_user_id" + ] + } + }, + "policies": {}, + "checkConstraints": { + "notion_mirrors_last_status_check": { + "name": "notion_mirrors_last_status_check", + "value": "\"last_status\" in ('ok', 'partial', 'failed')" + } + }, + "isRLSEnabled": false + } + }, + "enums": {}, + "schemas": {}, + "sequences": {}, + "roles": {}, + "policies": {}, + "views": {}, + "_meta": { + "columns": {}, + "schemas": {}, + "tables": {} + } +} \ No newline at end of file diff --git a/v5/src/lib/db/migrations/meta/_journal.json b/v5/src/lib/db/migrations/meta/_journal.json index b28ce2b..4323ed8 100644 --- a/v5/src/lib/db/migrations/meta/_journal.json +++ b/v5/src/lib/db/migrations/meta/_journal.json @@ -50,6 +50,13 @@ "when": 1790142683628, "tag": "0006_research_ledger", "breakpoints": true + }, + { + "idx": 7, + "version": "7", + "when": 1790163107556, + "tag": "0007_notion_mirror", + "breakpoints": true } ] } \ No newline at end of file diff --git a/v5/src/lib/db/schema/index.ts b/v5/src/lib/db/schema/index.ts index 89e6536..be20c67 100644 --- a/v5/src/lib/db/schema/index.ts +++ b/v5/src/lib/db/schema/index.ts @@ -3,8 +3,9 @@ * group; this module is what `drizzle.config.ts` points at and what * `drizzle()` receives as its `schema`, so relational queries see every table. * - * `pending_tools` arrived with Phase 6 (migration `0005`); the Notion mirror - * tables (Phase 8) are still to come, with their own migration. + * `pending_tools` arrived with Phase 6 (migration `0005`, and the research + * ledger in `0006`); the Notion mirror's `notion_mirrors` and `mirror_pages` + * with Phase 8 (migration `0007`). * * `auth.ts` is exported first because `helpers.ts` — which every other table * uses for `created_by` / `updated_by` — references `user.id`. @@ -21,3 +22,4 @@ export * from "./feedback.ts"; export * from "./projects.ts"; export * from "./audit.ts"; export * from "./pending-tools.ts"; +export * from "./mirror.ts"; diff --git a/v5/src/lib/db/schema/mirror.test.ts b/v5/src/lib/db/schema/mirror.test.ts new file mode 100644 index 0000000..feaf051 --- /dev/null +++ b/v5/src/lib/db/schema/mirror.test.ts @@ -0,0 +1,156 @@ +// @vitest-environment node +import { eq, sql } from "drizzle-orm"; +import { expectViolation } from "../../../../test/db"; +import { createPgliteDb } from "../pglite"; +import { rawRows } from "../raw"; +import { user } from "./auth"; +import { byteaToUint8Array, mirrorPages, notionMirrors } from "./mirror"; +import type { Db } from "../types"; + +/** + * Migration `0007` against a real (in-process) Postgres (spec §4.12). Each + * assertion is something only the database can get wrong: a CHECK, a unique + * owner, two cascades, a byte-exact bytea and the hand-appended trigger. + */ +describe("notion_mirrors and mirror_pages", () => { + let db: Db; + + beforeAll(async () => { + db = await createPgliteDb(); + }); + + async function insertUser(): Promise { + const id = `u-${Math.random().toString(36).slice(2)}`; + await db.insert(user).values({ id, name: "Mirror Owner", email: `${id}@cornell.edu` }); + return id; + } + + async function insertMirror(ownerUserId: string, overrides: Partial = {}) { + const [row] = await db + .insert(notionMirrors) + .values({ ownerUserId, parentPageId: "0f5e4a3c-1111-2222-3333-444455556666", ...overrides }) + .returning(); + return row; + } + + it("defaults the mapping to an empty object and the token to disconnected", async () => { + const row = await insertMirror(await insertUser()); + expect(row.mapping).toEqual({}); + expect(row.mappingGeneration).toBe(0); + expect(row.tokenCiphertext).toBeNull(); + expect(row.lastStatus).toBeNull(); + }); + + it("refuses a status outside the vocabulary", async () => { + const owner = await insertUser(); + await expectViolation( + insertMirror(owner, { lastStatus: "OK" }), + /notion_mirrors_last_status_check/ + ); + }); + + it("refuses an entity outside the vocabulary", async () => { + const mirror = await insertMirror(await insertUser()); + await expectViolation( + db.insert(mirrorPages).values({ + mirrorId: mirror.id, + entity: "maintenance_logs", + entityId: crypto.randomUUID(), + notionPageId: "page-1", + }), + /mirror_pages_entity_check/ + ); + }); + + it("allows one mirror per owner", async () => { + const owner = await insertUser(); + await insertMirror(owner); + await expectViolation(insertMirror(owner), /notion_mirrors_owner_user_id_unique/); + }); + + it("keys a page on (mirror, entity, entity id)", async () => { + const mirror = await insertMirror(await insertUser()); + const entityId = crypto.randomUUID(); + await db.insert(mirrorPages).values({ mirrorId: mirror.id, entity: "tools", entityId, notionPageId: "p-1" }); + // The same row under another entity is a different page. + await db.insert(mirrorPages).values({ mirrorId: mirror.id, entity: "units", entityId, notionPageId: "p-2" }); + await expectViolation( + db.insert(mirrorPages).values({ mirrorId: mirror.id, entity: "tools", entityId, notionPageId: "p-3" }), + /mirror_pages_mirror_id_entity_entity_id_pk/ + ); + }); + + it("deletes a mirror's pages with the mirror", async () => { + const mirror = await insertMirror(await insertUser()); + await db.insert(mirrorPages).values({ + mirrorId: mirror.id, + entity: "tools", + entityId: crypto.randomUUID(), + notionPageId: "p-1", + }); + + await db.delete(notionMirrors).where(eq(notionMirrors.id, mirror.id)); + + const left = await db.select().from(mirrorPages).where(eq(mirrorPages.mirrorId, mirror.id)); + expect(left).toHaveLength(0); + }); + + it("deletes the mirror, and its token, with its owner", async () => { + const owner = await insertUser(); + const mirror = await insertMirror(owner, { tokenCiphertext: new Uint8Array([1, 2, 3]) }); + + await db.delete(user).where(eq(user.id, owner)); + + const left = await db.select().from(notionMirrors).where(eq(notionMirrors.id, mirror.id)); + expect(left).toHaveLength(0); + }); + + it("round-trips a bytea byte for byte", async () => { + const bytes = new Uint8Array(256); + for (let i = 0; i < bytes.length; i += 1) bytes[i] = i; + const mirror = await insertMirror(await insertUser(), { tokenCiphertext: bytes }); + + const [read] = await db + .select({ token: notionMirrors.tokenCiphertext }) + .from(notionMirrors) + .where(eq(notionMirrors.id, mirror.id)); + expect(read.token).toBeInstanceOf(Uint8Array); + expect(Array.from(read.token as Uint8Array)).toEqual(Array.from(bytes)); + + const [length] = await rawRows<{ n: number }>( + db, + sql`select octet_length(token_ciphertext) as n from notion_mirrors where id = ${mirror.id}` + ); + expect(Number(length.n)).toBe(256); + }); + + it("maintains updated_at from the trigger", async () => { + const mirror = await insertMirror(await insertUser()); + + await new Promise((resolve) => setTimeout(resolve, 5)); + await db.update(notionMirrors).set({ parentPageTitle: "MakerLab Tools — mirror" }).where(eq(notionMirrors.id, mirror.id)); + + const [after] = await db.select().from(notionMirrors).where(eq(notionMirrors.id, mirror.id)); + expect(after.updatedAt.getTime()).toBeGreaterThan(mirror.updatedAt.getTime()); + + const triggers = await rawRows<{ tgname: string }>( + db, + sql`select tgname from pg_trigger where tgname = 'notion_mirrors_set_updated_at'` + ); + expect(triggers).toHaveLength(1); + }); +}); + +describe("byteaToUint8Array", () => { + it("normalises every shape a driver may return", () => { + expect(Array.from(byteaToUint8Array(new Uint8Array([1, 255])))).toEqual([1, 255]); + expect(Array.from(byteaToUint8Array(Buffer.from([1, 255])))).toEqual([1, 255]); + expect(Array.from(byteaToUint8Array("\\x01ff"))).toEqual([1, 255]); + expect(Array.from(byteaToUint8Array(new Uint8Array([9, 1, 255]).subarray(1)))).toEqual([1, 255]); + }); + + it("refuses what is not bytes", () => { + expect(() => byteaToUint8Array("\\xzz")).toThrow(TypeError); + expect(() => byteaToUint8Array(42)).toThrow(TypeError); + }); +}); diff --git a/v5/src/lib/db/schema/mirror.ts b/v5/src/lib/db/schema/mirror.ts new file mode 100644 index 0000000..eda6551 --- /dev/null +++ b/v5/src/lib/db/schema/mirror.ts @@ -0,0 +1,144 @@ +import { + customType, + integer, + jsonb, + pgTable, + primaryKey, + text, + timestamp, + uuid, +} from "drizzle-orm/pg-core"; +import type { MirrorLastError, MirrorMapping } from "../../mirror/types.ts"; +import { user } from "./auth.ts"; +import { inListCheck, timestamps } from "./helpers.ts"; +import { MIRROR_ENTITY, MIRROR_STATUS } from "./vocabulary.ts"; + +/** + * The Notion mirror (spec §3.8, §4.12; migration `0007`). + * + * Relative imports with `.ts` extensions: the mirror's workflow steps load the + * schema from an esbuild bundle under plain Node. + */ + +/** + * Whatever a driver hands back for a `bytea`, as a `Uint8Array`. + * + * PGlite returns a `Uint8Array`; the Neon (node-postgres) path a `Buffer`, + * which is one already but is copied so callers never hold a view onto a + * pooled buffer; and a driver that does not parse the type returns Postgres' + * hex text form, `\x0a1b…`. Raw-SQL readers bypass `fromDriver`, so they call + * this themselves. + */ +export function byteaToUint8Array(value: unknown): Uint8Array { + if (value instanceof Uint8Array) return new Uint8Array(value); + if (typeof value === "string") { + const hex = value.startsWith("\\x") ? value.slice(2) : value; + if (hex.length % 2 !== 0 || !/^[0-9a-fA-F]*$/.test(hex)) { + throw new TypeError("bytea: not a hex string"); + } + const bytes = new Uint8Array(hex.length / 2); + for (let i = 0; i < bytes.length; i += 1) { + bytes[i] = parseInt(hex.slice(i * 2, i * 2 + 2), 16); + } + return bytes; + } + if (value instanceof ArrayBuffer) return new Uint8Array(value.slice(0)); + if (ArrayBuffer.isView(value)) { + return new Uint8Array(value.buffer.slice(value.byteOffset, value.byteOffset + value.byteLength)); + } + throw new TypeError("bytea: unexpected driver value"); +} + +/** + * A `bytea` column as a `Uint8Array`. Written as a `Buffer`, which is a + * `Uint8Array` to PGlite and the one binary type every node-postgres version + * serialises as bytea rather than as text. + */ +export const bytea = customType<{ data: Uint8Array; driverData: Uint8Array | string }>({ + dataType() { + return "bytea"; + }, + toDriver(value) { + return Buffer.from(value.buffer, value.byteOffset, value.byteLength); + }, + fromDriver(value) { + return byteaToUint8Array(value); + }, +}); + +/** + * One mirror per admin who has set one up (§4.12). + * + * - `owner_user_id` is **unique and cascades**: a mirror is one person's, and + * deleting the person must delete their token with them. + * - `token_ciphertext` is the Notion token under AES-256-GCM + * (`mirror/token-crypto.ts`). **Null means disconnected**: Disconnect forgets + * the token and keeps the mapping and `mirror_pages`, so reconnecting the + * same workspace updates the same pages instead of duplicating them. (§4.12 + * typed it not null; the nullable column is what makes that possible.) + * - `running_since` is the overlap guard; `push_requested_at` and + * `sync_requested_at` are the coalescing and Sync-now claims; `last_run_at` + * is when the last push finished, whatever its result, beside + * `last_synced_at`, which only moves when everything up to it was pushed. + * - `last_error` is a {@link MirrorLastError}: a code the page translates and a + * scrubbed detail, never a token or an email. + * - `mapping_generation` goes up by one whenever the mapping changes or an + * entity's pages are forgotten (`setMirrorMapping`, `resetMirrorEntities`). + * A push carries the generation it claimed under, and neither advances + * `last_synced_at` nor records a page once it has moved: that push was + * working from a mapping that is no longer true. + * + * The backup export nulls `token_ciphertext` (`cron/backup-policy.ts`). + */ +export const notionMirrors = pgTable( + "notion_mirrors", + { + id: uuid("id").primaryKey().defaultRandom(), + ownerUserId: text("owner_user_id") + .notNull() + .unique() + .references(() => user.id, { onDelete: "cascade" }), + tokenCiphertext: bytea("token_ciphertext"), + parentPageId: text("parent_page_id").notNull(), + parentPageTitle: text("parent_page_title"), + mapping: jsonb("mapping").$type().notNull().default({}), + mappingGeneration: integer("mapping_generation").notNull().default(0), + pausedAt: timestamp("paused_at", { withTimezone: true }), + runningSince: timestamp("running_since", { withTimezone: true }), + pushRequestedAt: timestamp("push_requested_at", { withTimezone: true }), + syncRequestedAt: timestamp("sync_requested_at", { withTimezone: true }), + lastSyncedAt: timestamp("last_synced_at", { withTimezone: true }), + lastRunAt: timestamp("last_run_at", { withTimezone: true }), + lastStatus: text("last_status"), + lastError: jsonb("last_error").$type(), + ...timestamps(), + }, + () => [inListCheck("notion_mirrors_last_status_check", "last_status", MIRROR_STATUS)] +); + +/** + * Which Notion page mirrors which app row (§4.12). Keyed on the mirror, the + * entity and the row, so each mirror keeps its own pages and deleting a mirror + * deletes its map. + * + * `source_updated_at` is the source row's `updated_at` as it was pushed, + * written from the text the push selected so it keeps Postgres' microseconds. + * No `updated_at` trigger: `pushed_at` is set by the push on every write. + */ +export const mirrorPages = pgTable( + "mirror_pages", + { + mirrorId: uuid("mirror_id") + .notNull() + .references(() => notionMirrors.id, { onDelete: "cascade" }), + entity: text("entity").notNull(), + entityId: uuid("entity_id").notNull(), + notionPageId: text("notion_page_id").notNull(), + pushedAt: timestamp("pushed_at", { withTimezone: true }).notNull().defaultNow(), + sourceUpdatedAt: timestamp("source_updated_at", { withTimezone: true }), + }, + (t) => [ + primaryKey({ columns: [t.mirrorId, t.entity, t.entityId] }), + inListCheck("mirror_pages_entity_check", "entity", MIRROR_ENTITY), + ] +); diff --git a/v5/src/lib/db/schema/vocabulary.ts b/v5/src/lib/db/schema/vocabulary.ts index 99e563d..1ce6b80 100644 --- a/v5/src/lib/db/schema/vocabulary.ts +++ b/v5/src/lib/db/schema/vocabulary.ts @@ -96,6 +96,34 @@ export type AttachmentOwner = (typeof ATTACHMENT_OWNER)[number]; export const ATTACHMENT_ACCESS = ["public", "private"] as const; export type AttachmentAccess = (typeof ATTACHMENT_ACCESS)[number]; +/** + * What a Notion mirror pushes, one Notion database each (spec §3.8, §4.12). + * + * **Declared in dependency order, and that order is the push order**: a tool + * page relates to its category and location, a unit to its tool, and so on, so + * a page is only ever created after the pages it points at. `maintenance` is + * `maintenance_logs` in Postgres; the short name is what the admin page shows + * and what `mirror_pages.entity` stores. + */ +export const MIRROR_ENTITY = [ + "categories", + "locations", + "tools", + "units", + "resources", + "maintenance", + "projects", +] as const; +export type MirrorEntity = (typeof MIRROR_ENTITY)[number]; + +/** + * The result of a mirror's last push (spec §3.8 "Status"): everything pushed, + * some rows failed (`last_synced_at` does not advance, so they are retried), or + * nothing could be pushed at all. + */ +export const MIRROR_STATUS = ["ok", "partial", "failed"] as const; +export type MirrorStatus = (typeof MIRROR_STATUS)[number]; + /** True when `value` is one of `list`; narrows the type. */ export function isOneOf( list: T, diff --git a/v5/src/lib/import/blob-uploader.ts b/v5/src/lib/import/blob-uploader.ts index 8cba2c5..13d0ff3 100644 --- a/v5/src/lib/import/blob-uploader.ts +++ b/v5/src/lib/import/blob-uploader.ts @@ -1,4 +1,6 @@ import { put } from "@vercel/blob"; +import { createLocalBlobBackend } from "../blob-local.ts"; +import { blobMode } from "../blob-mode.ts"; import type { BlobUploader } from "./files.ts"; /** @@ -6,6 +8,10 @@ import type { BlobUploader } from "./files.ts"; * `BLOB_READ_WRITE_TOKEN`. A random suffix is always added so a pathname can * never be guessed from a filename, and so two files with the same name under * one owner never collide. + * + * The Notion import calls this directly and so still insists on a token: its + * rows go to `DATABASE_URL`, and a shared database must not end up pointing at + * files on one person's laptop. */ export function createVercelBlobUploader(): BlobUploader { if (!process.env.BLOB_READ_WRITE_TOKEN) { @@ -22,3 +28,32 @@ export function createVercelBlobUploader(): BlobUploader { }, }; } + +/** The same contract against `.blob-data/` (see `blob-local.ts`). */ +export function createLocalBlobUploader(): BlobUploader { + const disk = createLocalBlobBackend(); + return { + put(pathname, body, options) { + return disk.put(pathname, body, { + access: options.access, + contentType: options.contentType, + addRandomSuffix: true, + }); + }, + }; +} + +/** + * Whichever uploader `blobMode()` allows, or null when there is no store — + * the step-code counterpart of `lib/blob.ts`'s `getBlobStore()`. + */ +export function createBlobUploader(): BlobUploader | null { + switch (blobMode()) { + case "vercel": + return createVercelBlobUploader(); + case "local": + return createLocalBlobUploader(); + default: + return null; + } +} diff --git a/v5/src/lib/intake/approve.test.ts b/v5/src/lib/intake/approve.test.ts index c0196db..01e4faf 100644 --- a/v5/src/lib/intake/approve.test.ts +++ b/v5/src/lib/intake/approve.test.ts @@ -19,6 +19,19 @@ vi.mock("../data/audit", async (importOriginal) => { }; }); +// The mirror trigger has its own tests; here it is only asked whether it was +// called — after a committed approval, never after a refused one. +const mirror = vi.hoisted(() => ({ requestMirrorPush: vi.fn() })); + +vi.mock("../mirror/trigger", () => ({ requestMirrorPush: mirror.requestMirrorPush })); + +// The manual archive is mocked one layer down, at the module that calls the +// Workflow SDK's `start()`, so the real trigger's never-throw rule is what the +// start-failure case exercises. +const manuals = vi.hoisted(() => ({ startManualArchive: vi.fn() })); + +vi.mock("../manuals/start", () => ({ startManualArchive: manuals.startManualArchive })); + import { revalidateTag } from "next/cache"; import { eq } from "drizzle-orm"; import { seedUser } from "../../../test/utils/session"; @@ -33,7 +46,7 @@ import { type ApprovalFields, } from "../data/pending-tools"; import { getDb, resetDbForTests } from "../db/client"; -import { auditEvents, tools, units } from "../db/schema/index"; +import { auditEvents, resources, tools, units } from "../db/schema/index"; import type { Db } from "../db/types"; import type { ResearchResult } from "../research/result"; import { CATALOG_TAG } from "../revalidate"; @@ -55,6 +68,8 @@ let approver: string; beforeEach(async () => { vi.stubEnv("DATABASE_URL", ""); vi.mocked(revalidateTag).mockClear(); + mirror.requestMirrorPush.mockReset().mockResolvedValue(undefined); + manuals.startManualArchive.mockReset().mockResolvedValue(true); audit.failing = false; db = await getDb(); approver = (await seedUser({ email: "luis@cornell.edu", role: "admin" })).id; @@ -301,3 +316,79 @@ describe("addUnitAndRecord", () => { expect(vi.mocked(revalidateTag)).not.toHaveBeenCalled(); }); }); + +describe("the Notion mirror (§3.8 trigger 1)", () => { + it("asks for a push once an approval has committed, published or draft", async () => { + const published = await researchedItem(); + expect((await approveAndRecord({ userId: approver }, { id: published, publish: true, fields: fields() })).ok).toBe(true); + expect(mirror.requestMirrorPush).toHaveBeenCalledTimes(1); + + const draft = await researchedItem(); + expect( + (await approveAndRecord({ userId: approver }, { id: draft, publish: false, fields: fields({ serialNumber: "P1S-002" }) })).ok + ).toBe(true); + expect(mirror.requestMirrorPush).toHaveBeenCalledTimes(2); + }); + + it("asks for a push when a unit is added", async () => { + const id = await unitItem("F4-MIRROR-1"); + expect((await addUnitAndRecord({ userId: approver }, { id })).ok).toBe(true); + expect(mirror.requestMirrorPush).toHaveBeenCalledTimes(1); + }); + + it("passes the caller's database handle through", async () => { + const id = await unitItem("F4-MIRROR-2"); + await addUnitAndRecord({ userId: approver }, { id }, { db }); + expect(mirror.requestMirrorPush).toHaveBeenCalledWith({ db }); + }); + + it("asks for nothing when the approval is refused", async () => { + const low = await researchedItem(research(LOW)); + expect((await approveAndRecord({ userId: approver }, { id: low, publish: true, fields: fields() })).ok).toBe(false); + + const first = await unitItem("F4-MIRROR-SAME"); + const second = await unitItem("F4-MIRROR-SAME"); + expect((await addUnitAndRecord({ userId: approver }, { id: first })).ok).toBe(true); + mirror.requestMirrorPush.mockClear(); + expect((await addUnitAndRecord({ userId: approver }, { id: second })).ok).toBe(false); + + expect(mirror.requestMirrorPush).not.toHaveBeenCalled(); + }); +}); + +describe("the manual archive", () => { + it("starts an archive run for the resources the approval created, after the commit", async () => { + const id = await researchedItem(); + const result = await approveAndRecord({ userId: approver }, { id, publish: true, fields: fields() }); + if (!result.ok) throw new Error("approval refused"); + + const created = await db.select({ id: resources.id }).from(resources).where(eq(resources.toolId, result.toolId)); + expect(created).toHaveLength(1); + expect(manuals.startManualArchive).toHaveBeenCalledTimes(1); + expect(manuals.startManualArchive).toHaveBeenCalledWith([created[0].id]); + }); + + it("still approves when the run cannot be started, with no warning and nothing rolled back", async () => { + const error = vi.spyOn(console, "error").mockImplementation(() => {}); + manuals.startManualArchive.mockRejectedValue(new Error("workflow runtime unavailable")); + const id = await researchedItem(); + + const result = await approveAndRecord({ userId: approver }, { id, publish: true, fields: fields() }); + + expect(result).toMatchObject({ ok: true, published: true }); + expect(result).not.toHaveProperty("warning"); + if (!result.ok) throw new Error("unreachable"); + expect(await db.select().from(tools).where(eq(tools.id, result.toolId))).toHaveLength(1); + expect(error).toHaveBeenCalledWith(expect.stringContaining("[manuals]")); + }); + + it("starts nothing when the approval is refused, or when it created no resources", async () => { + const low = await researchedItem(research(LOW)); + expect((await approveAndRecord({ userId: approver }, { id: low, publish: true, fields: fields() })).ok).toBe(false); + + const bare = await researchedItem(research({ resources: [] })); + expect((await approveAndRecord({ userId: approver }, { id: bare, publish: true, fields: fields() })).ok).toBe(true); + + expect(manuals.startManualArchive).not.toHaveBeenCalled(); + }); +}); diff --git a/v5/src/lib/intake/approve.ts b/v5/src/lib/intake/approve.ts index 1435c93..fc2b0f7 100644 --- a/v5/src/lib/intake/approve.ts +++ b/v5/src/lib/intake/approve.ts @@ -9,6 +9,8 @@ import { } from "../data/pending-tools"; import { getDb } from "../db/client"; import type { Db } from "../db/types"; +import { requestManualArchive } from "../manuals/trigger"; +import { requestMirrorPush } from "../mirror/trigger"; import { invalidateCatalog } from "../revalidate"; /** @@ -30,6 +32,15 @@ import { invalidateCatalog } from "../revalidate"; * 2. **The catalogue cache.** `invalidateCatalog()`, for a draft too — the * inventory table and `catalog.view_drafts` read it, and a draft is one * click from published. + * 3. **The Notion mirror.** `requestMirrorPush()` (§3.8 trigger 1: "Approving + * a tool … calls `requestMirrorPush()`"), for a draft too, since the mirror + * carries every tool with a Published checkbox. It never throws and costs + * one query when nobody has a mirror; the push itself runs minutes later + * in a workflow, so Notion being down cannot touch an approval. + * 4. **The manual archive.** `requestManualArchive()` with the resources the + * approval created, so each manual PDF is copied into Blob before the + * manufacturer moves it. Also a workflow, also never throws: a run that + * could not be started is logged and left to the nightly backfill. * * **A lost audit event is a warning on a success, never a failure.** The tool * exists by the time the event is written; answering `{ ok: false }` would tell @@ -75,8 +86,8 @@ export interface IntakeApprovalOptions { * * The write is one transaction in the data layer; everything here runs only * once it has committed, and only as far as each step earns. A refusal stops - * at the write — nothing changed, so there is nothing to record and nothing - * stale to bust. + * at the write — nothing changed, so there is nothing to record, nothing + * stale to bust and nothing to mirror. */ export async function approveAndRecord( approver: IntakeApprover, @@ -124,6 +135,8 @@ export async function approveAndRecord( : true; invalidateCatalog(); + await requestMirrorPush({ db: options.db }); + await requestManualArchive(approved.resourceIds); return { ok: true, @@ -158,8 +171,10 @@ export async function addUnitAndRecord( if (!added.ok) return { ok: false, error: added.reason }; const published = added.published; - // A new unit changes what the tool page says about availability. + // A new unit changes what the tool page says about availability — and what + // the mirror's Units database holds. Neither can throw. invalidateCatalog(); + await requestMirrorPush({ db }); const recorded = await record( { diff --git a/v5/src/lib/manuals/archive.test.ts b/v5/src/lib/manuals/archive.test.ts new file mode 100644 index 0000000..2579c69 --- /dev/null +++ b/v5/src/lib/manuals/archive.test.ts @@ -0,0 +1,314 @@ +// @vitest-environment node +import { eq } from "drizzle-orm"; +import { http, HttpResponse } from "msw"; +import { server } from "../../../test/msw/server"; +import { manualSourceKey } from "../data/manual-archives"; +import { createPgliteDb } from "../db/pglite"; +import { attachments, resources, tools } from "../db/schema/index"; +import type { Db } from "../db/types"; +import type { BlobUploader } from "../import/files"; +import { + archiveManual, + filenameFromDisposition, + looksLikePdf, + MAX_MANUAL_BYTES, +} from "./archive"; + +/** + * The manual archive against PGlite, with the manufacturer answered by MSW and + * Blob by an in-memory uploader. No environment, no network. + */ + +const PDF = new TextEncoder().encode("%PDF-1.7\n1 0 obj\n<<>>\nendobj\n%%EOF\n"); + +let db: Db; +let toolId: string; +let puts: Array<{ pathname: string; access: string; contentType?: string; bytes: number }>; +let uploader: BlobUploader; + +beforeAll(async () => { + db = await createPgliteDb(); +}); + +beforeEach(async () => { + vi.stubEnv("BLOB_READ_WRITE_TOKEN", ""); + vi.spyOn(console, "info").mockImplementation(() => {}); + vi.spyOn(console, "warn").mockImplementation(() => {}); + await db.delete(attachments); + await db.delete(tools); + const [tool] = await db.insert(tools).values({ slug: "p1s", name: "Bambu Lab P1S", published: true }).returning({ id: tools.id }); + toolId = tool.id; + puts = []; + let n = 0; + uploader = { + async put(pathname, body, options) { + n += 1; + puts.push({ pathname, access: options.access, contentType: options.contentType, bytes: body.byteLength }); + const stored = pathname.replace(/\.pdf$/, `-r${n}.pdf`); + return { pathname: stored, url: `https://blob.test/${stored}` }; + }, + }; +}); + +afterEach(() => { + vi.unstubAllEnvs(); + vi.restoreAllMocks(); +}); + +async function resource(values: Partial = {}): Promise { + const [row] = await db + .insert(resources) + .values({ toolId, title: "P1S manual", type: "Manual", url: "https://maker.test/p1s-manual.pdf", ...values }) + .returning({ id: resources.id }); + return row.id; +} + +function servePdf(url: string, headers: Record = {}) { + server.use( + http.get(url, () => + HttpResponse.arrayBuffer(PDF.slice().buffer, { headers: { "content-type": "application/pdf", ...headers } }) + ) + ); +} + +async function owned(resourceId: string) { + return db.select().from(attachments).where(eq(attachments.ownerId, resourceId)); +} + +describe("archiveManual", () => { + it("archives a manual PDF into Blob as a public attachment owned by the resource", async () => { + const id = await resource(); + servePdf("https://maker.test/p1s-manual.pdf"); + + const result = await archiveManual(id, { db, uploader }); + + expect(result).toMatchObject({ status: "archived", reason: "archived", sizeBytes: PDF.byteLength }); + expect(puts).toEqual([ + { pathname: `manuals/${toolId}/${id}.pdf`, access: "public", contentType: "application/pdf", bytes: PDF.byteLength }, + ]); + const [row] = await owned(id); + expect(row).toMatchObject({ + ownerType: "resource", + ownerId: id, + access: "public", + contentType: "application/pdf", + sizeBytes: PDF.byteLength, + originalFilename: "p1s-manual.pdf", + sourceKey: manualSourceKey(id, "https://maker.test/p1s-manual.pdf"), + publicUrl: `https://blob.test/manuals/${toolId}/${id}-r1.pdf`, + }); + // The manufacturer's link stays on the resource. + const [kept] = await db.select({ url: resources.url }).from(resources).where(eq(resources.id, id)); + expect(kept.url).toBe("https://maker.test/p1s-manual.pdf"); + }); + + it("takes the file name from Content-Disposition when there is one", async () => { + const id = await resource({ url: "https://maker.test/download?id=42" }); + servePdf("https://maker.test/download", { "content-disposition": 'attachment; filename="P1S Manual EN.pdf"' }); + + expect((await archiveManual(id, { db, uploader })).status).toBe("archived"); + expect((await owned(id))[0].originalFilename).toBe("P1S Manual EN.pdf"); + }); + + it("refuses an HTML product page, stores nothing, and does not retry it", async () => { + const id = await resource({ url: "https://maker.test/products/p1s" }); + server.use(http.get("https://maker.test/products/p1s", () => HttpResponse.html("Buy now"))); + + expect(await archiveManual(id, { db, uploader })).toEqual({ status: "failed", reason: "not_pdf", transient: false }); + expect(puts).toEqual([]); + expect(await owned(id)).toEqual([]); + }); + + it("refuses markup served as octet-stream: the bytes have to be a PDF", async () => { + const id = await resource({ url: "https://maker.test/manual.bin" }); + server.use( + http.get("https://maker.test/manual.bin", () => + HttpResponse.arrayBuffer(new TextEncoder().encode("

hi

").buffer, { + headers: { "content-type": "application/octet-stream" }, + }) + ) + ); + + expect(await archiveManual(id, { db, uploader })).toMatchObject({ status: "failed", reason: "not_pdf" }); + expect(puts).toEqual([]); + }); + + it("skips a non-Manual link that turns out not to be a PDF", async () => { + const id = await resource({ type: "SOP", url: "https://maker.test/sop" }); + server.use(http.get("https://maker.test/sop", () => HttpResponse.html("

SOP

"))); + + expect(await archiveManual(id, { db, uploader })).toEqual({ status: "skipped", reason: "not_pdf" }); + }); + + it("archives a non-Manual resource whose link is a PDF", async () => { + const id = await resource({ type: "Safety", url: "https://maker.test/safety.pdf" }); + servePdf("https://maker.test/safety.pdf"); + + expect((await archiveManual(id, { db, uploader })).status).toBe("archived"); + }); + + it("refuses a file declared larger than 25 MB without reading it", async () => { + const id = await resource(); + servePdf("https://maker.test/p1s-manual.pdf", { "content-length": String(MAX_MANUAL_BYTES + 1) }); + + expect(await archiveManual(id, { db, uploader })).toEqual({ status: "failed", reason: "too_large", transient: false }); + expect(puts).toEqual([]); + }); + + it("refuses a body that grows past 25 MB with no length declared", async () => { + const id = await resource(); + const chunk = new Uint8Array(1024 * 1024); + chunk.set(PDF); + server.use( + http.get("https://maker.test/p1s-manual.pdf", () => { + let sent = 0; + const stream = new ReadableStream({ + pull(controller) { + if (sent > MAX_MANUAL_BYTES) return controller.close(); + sent += chunk.byteLength; + controller.enqueue(chunk); + }, + }); + return new HttpResponse(stream, { headers: { "content-type": "application/pdf" } }); + }) + ); + + expect(await archiveManual(id, { db, uploader })).toMatchObject({ status: "failed", reason: "too_large" }); + expect(puts).toEqual([]); + }); + + it("marks a 5xx transient and a 404 not", async () => { + const flaky = await resource({ url: "https://maker.test/flaky.pdf" }); + const gone = await resource({ url: "https://maker.test/gone.pdf" }); + server.use( + http.get("https://maker.test/flaky.pdf", () => new HttpResponse(null, { status: 503 })), + http.get("https://maker.test/gone.pdf", () => new HttpResponse(null, { status: 404 })) + ); + + expect(await archiveManual(flaky, { db, uploader })).toEqual({ + status: "failed", + reason: "http_error", + transient: true, + httpStatus: 503, + }); + expect(await archiveManual(gone, { db, uploader })).toEqual({ + status: "failed", + reason: "http_error", + transient: false, + httpStatus: 404, + }); + }); + + it("marks a network failure transient", async () => { + const id = await resource(); + server.use(http.get("https://maker.test/p1s-manual.pdf", () => HttpResponse.error())); + + expect(await archiveManual(id, { db, uploader })).toEqual({ status: "failed", reason: "download_failed", transient: true }); + }); + + it("skips a resource already archived from this link, without a download", async () => { + const id = await resource(); + servePdf("https://maker.test/p1s-manual.pdf"); + expect((await archiveManual(id, { db, uploader })).status).toBe("archived"); + + // No handler this time: a request would fail the test (onUnhandledRequest). + server.resetHandlers(); + expect(await archiveManual(id, { db, uploader })).toEqual({ status: "skipped", reason: "already_archived" }); + expect(puts).toHaveLength(1); + }); + + it("skips a resource that already holds an uploaded or imported PDF", async () => { + const id = await resource(); + await db.insert(attachments).values({ + ownerType: "resource", + ownerId: id, + blobPathname: "resources/x/manual.pdf", + access: "public", + publicUrl: "https://blob.test/resources/x/manual.pdf", + contentType: "application/pdf", + sourceKey: "notion-file-1", + }); + + expect(await archiveManual(id, { db, uploader })).toEqual({ status: "skipped", reason: "has_file" }); + }); + + it("archives the new link after an edit and releases the stale copy to the sweep", async () => { + const id = await resource({ url: "https://maker.test/old.pdf" }); + servePdf("https://maker.test/old.pdf"); + servePdf("https://maker.test/new.pdf"); + expect((await archiveManual(id, { db, uploader })).status).toBe("archived"); + + await db.update(resources).set({ url: "https://maker.test/new.pdf" }).where(eq(resources.id, id)); + expect((await archiveManual(id, { db, uploader })).status).toBe("archived"); + + const rows = await owned(id); + expect(rows.map((row) => row.sourceKey)).toEqual([manualSourceKey(id, "https://maker.test/new.pdf")]); + const [released] = await db + .select({ ownerId: attachments.ownerId }) + .from(attachments) + .where(eq(attachments.sourceKey, manualSourceKey(id, "https://maker.test/old.pdf"))); + expect(released.ownerId).toBeNull(); + }); + + it("archives the same manufacturer URL on two resources, one copy each", async () => { + const other = (await db.insert(tools).values({ slug: "p1s-2", name: "P1S (lab 2)" }).returning({ id: tools.id }))[0].id; + const a = await resource(); + const b = await resource({ toolId: other }); + servePdf("https://maker.test/p1s-manual.pdf"); + + expect((await archiveManual(a, { db, uploader })).status).toBe("archived"); + expect((await archiveManual(b, { db, uploader })).status).toBe("archived"); + expect((await owned(a))[0].sourceKey).toBe(manualSourceKey(a, "https://maker.test/p1s-manual.pdf")); + expect((await owned(b))[0].sourceKey).toBe(manualSourceKey(b, "https://maker.test/p1s-manual.pdf")); + expect(puts.map((p) => p.pathname)).toEqual([`manuals/${toolId}/${a}.pdf`, `manuals/${other}/${b}.pdf`]); + }); + + it("skips when Blob is not configured, before any download", async () => { + const id = await resource(); + + expect(await archiveManual(id, { db })).toEqual({ status: "skipped", reason: "blob_not_configured" }); + }); + + it("skips a resource with no link, and an id that is not one", async () => { + const id = await resource({ url: null }); + + expect(await archiveManual(id, { db, uploader })).toEqual({ status: "skipped", reason: "no_url" }); + expect(await archiveManual("not-a-uuid", { db, uploader })).toEqual({ status: "skipped", reason: "not_found" }); + expect(await archiveManual(crypto.randomUUID(), { db, uploader })).toEqual({ status: "skipped", reason: "not_found" }); + }); + + it("reports a failed Blob write as transient and records nothing", async () => { + const id = await resource(); + servePdf("https://maker.test/p1s-manual.pdf"); + const broken: BlobUploader = { put: async () => Promise.reject(new Error("blob down")) }; + + expect(await archiveManual(id, { db, uploader: broken })).toEqual({ status: "failed", reason: "upload_failed", transient: true }); + expect(await owned(id)).toEqual([]); + }); + + it("never logs the link's path or query", async () => { + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + const id = await resource({ url: "https://maker.test/signed/manual.pdf?token=s3cr3t" }); + server.use(http.get("https://maker.test/signed/manual.pdf", () => new HttpResponse(null, { status: 403 }))); + + await archiveManual(id, { db, uploader }); + const logged = warn.mock.calls.flat().join(" "); + expect(logged).toContain("host=maker.test"); + expect(logged).not.toContain("s3cr3t"); + expect(logged).not.toContain("/signed/"); + }); +}); + +describe("looksLikePdf / filenameFromDisposition", () => { + it("finds the magic bytes behind a little leading junk", () => { + const bytes = new Uint8Array([0x0a, 0x0a, ...PDF]); + expect(looksLikePdf(bytes, "application/octet-stream")).toBe(true); + expect(looksLikePdf(new TextEncoder().encode(""), "application/pdf")).toBe(false); + }); + + it("reads both forms of the header", () => { + expect(filenameFromDisposition("attachment; filename*=UTF-8''Manual%20v2.pdf")).toBe("Manual v2.pdf"); + expect(filenameFromDisposition('inline; filename="a/b/c.pdf"')).toBe("c.pdf"); + expect(filenameFromDisposition(null)).toBeNull(); + }); +}); diff --git a/v5/src/lib/manuals/archive.ts b/v5/src/lib/manuals/archive.ts new file mode 100644 index 0000000..b42b697 --- /dev/null +++ b/v5/src/lib/manuals/archive.ts @@ -0,0 +1,408 @@ +import { eq } from "drizzle-orm"; +import { claimAttachments, createAttachment } from "../data/attachments.ts"; +import { + listResourcePdfs, + manualSourceKey, + MAX_ARCHIVABLE_URL_LENGTH, + isManualArchiveKey, + releaseStaleManualArchives, +} from "../data/manual-archives.ts"; +import { isUuid } from "../data/uuid.ts"; +import { getDb } from "../db/client.ts"; +import { resources } from "../db/schema/index.ts"; +import type { Db } from "../db/types.ts"; +import { createBlobUploader } from "../import/blob-uploader.ts"; +import { safeFilename, type BlobUploader } from "../import/files.ts"; + +/** + * `archiveManual(resourceId)` — copy a resource's manual PDF into Blob, so the + * tool keeps its manual after the manufacturer moves or deletes it (link rot). + * + * The copy is an `attachments` row owned by the resource — public, + * `application/pdf`, keyed `manual::` (see + * `data/manual-archives.ts`). The resource keeps its own `url`, the + * manufacturer's link; the catalogue and the chat prefer the copy. + * + * **What is archived.** A resource with an http(s) link that is a Manual, or + * whose link turns out to be a PDF. The response must *be* a PDF — `%PDF-` in + * its first kilobyte, or `application/pdf` on a body that is not markup — so a + * manual link that points at an HTML product page is refused, not stored as a + * "manual". 30 seconds, 25 MB, and nothing larger is read. + * + * **It never throws for an expected failure.** Every outcome is a value: + * `archived`, `skipped` (nothing to do, or nothing it may do) or `failed` + * (tried and could not), with `transient` on a failure a retry could fix — a + * dropped connection, a timeout, a 5xx or a 429. The workflow step retries + * only those. What does throw is the database, which the step classifies. + * + * **Nothing secret is logged.** The one log line carries the resource id, the + * outcome and the link's host — never its path or query, which for a signed + * URL is the credential. + * + * Runs as a workflow step under plain Node: relative imports, and no + * `"server-only"` anywhere below it — which is why the Blob write goes through + * the import's uploader (`import/blob-uploader.ts`) rather than `lib/blob.ts`. + */ + +/** Longest a download may take, headers and body together. */ +export const MANUAL_FETCH_TIMEOUT_MS = 30_000; + +/** Largest manual archived. Bigger ones are refused, not truncated. */ +export const MAX_MANUAL_BYTES = 25 * 1024 * 1024; + +/** How far into the body `%PDF-` may appear (the PDF spec allows leading junk). */ +const PDF_MAGIC_WINDOW = 1024; + +const FETCH_USER_AGENT = "Mozilla/5.0 (compatible; MakerLabBot/1.0; manual archive)"; + +export type ArchiveSkipReason = + /** No such resource, or not a uuid. */ + | "not_found" + /** No http(s) link to copy from. */ + | "no_url" + /** A link too long to key (see `MAX_ARCHIVABLE_URL_LENGTH`). */ + | "url_too_long" + /** Already archived from this link. */ + | "already_archived" + /** The resource already holds a PDF somebody uploaded or the import copied. */ + | "has_file" + /** Not a Manual, and the link did not answer a PDF. */ + | "not_pdf" + /** No Blob store: no `BLOB_READ_WRITE_TOKEN`, and not local development (`blob-mode.ts`). */ + | "blob_not_configured"; + +export type ArchiveFailReason = + /** The request never got an answer: DNS, connection, timeout. */ + | "download_failed" + /** The host answered with an error status. */ + | "http_error" + /** A Manual whose link answered something other than a PDF — a product page, usually. */ + | "not_pdf" + /** Over {@link MAX_MANUAL_BYTES}. */ + | "too_large" + /** An empty body. */ + | "empty" + /** The Blob write failed. */ + | "upload_failed"; + +export type ArchiveManualResult = + | { status: "archived"; reason: "archived"; attachmentId: string; publicUrl: string; sizeBytes: number } + | { status: "skipped"; reason: ArchiveSkipReason } + | { status: "failed"; reason: ArchiveFailReason; transient: boolean; httpStatus?: number }; + +export interface ArchiveManualOptions { + /** A handle to use instead of {@link getDb} — tests pass an isolated one. */ + db?: Db; + /** The download. Tests pass one; MSW answers it. */ + fetchImpl?: typeof fetch; + /** Cancels the download along with the 30-second timeout. */ + signal?: AbortSignal; + /** + * The Blob write. Defaults to whatever `blobMode()` allows — Vercel Blob + * with a token, `.blob-data/` in local development; with no store and no + * uploader, the archive is skipped as `blob_not_configured`. + */ + uploader?: BlobUploader; +} + +export async function archiveManual( + resourceId: string, + options: ArchiveManualOptions = {} +): Promise { + const result = await archive(resourceId, options); + logOutcome(resourceId, result.outcome, result.host); + return result.outcome; +} + +interface Attempt { + outcome: ArchiveManualResult; + host?: string; +} + +async function archive(resourceId: string, options: ArchiveManualOptions): Promise { + if (!isUuid(resourceId)) return skip("not_found"); + const db = options.db ?? (await getDb()); + + const [resource] = await db + .select({ id: resources.id, toolId: resources.toolId, title: resources.title, type: resources.type, url: resources.url }) + .from(resources) + .where(eq(resources.id, resourceId)); + if (!resource) return skip("not_found"); + + const url = resource.url?.trim() ?? ""; + if (!/^https?:\/\//i.test(url)) return skip("no_url"); + const host = hostOf(url); + if (url.length > MAX_ARCHIVABLE_URL_LENGTH) return skip("url_too_long", host); + + const key = manualSourceKey(resource.id, url); + const held = await listResourcePdfs(db, resource.id); + if (held.some((pdf) => pdf.sourceKey === key)) return skip("already_archived", host); + if (held.some((pdf) => !isManualArchiveKey(pdf.sourceKey))) return skip("has_file", host); + + const uploader = options.uploader ?? createBlobUploader(); + if (!uploader) return skip("blob_not_configured", host); + + const isManual = (resource.type ?? "").trim().toLowerCase() === "manual"; + + const downloaded = await download(url, options); + if (!downloaded.ok) { + // A link that is not a Manual and does not answer a PDF is simply a link. + if (downloaded.reason === "not_pdf" && !isManual) return skip("not_pdf", host); + return { outcome: { status: "failed", ...downloaded.failure }, host }; + } + + let stored: { pathname: string; url: string }; + try { + stored = await uploader.put(`manuals/${resource.toolId ?? "unassigned"}/${resource.id}.pdf`, downloaded.bytes, { + access: "public", + contentType: "application/pdf", + }); + } catch { + return fail("upload_failed", true, host); + } + + const filename = downloaded.filename ?? filenameFromUrl(url) ?? `${safeFilename(resource.title, "manual")}.pdf`; + const file = { + blobPathname: stored.pathname, + access: "public" as const, + publicUrl: stored.url, + contentType: "application/pdf", + sizeBytes: downloaded.bytes.byteLength, + originalFilename: filename, + uploadedBy: null, + }; + + let attachmentId: string | null; + try { + attachmentId = await db.transaction(async (tx) => { + // Somebody else's run may have landed the same copy while this one + // downloaded (approval and the nightly backfill can overlap). Its row + // wins — never hold a transaction open across a 30-second download. + const now = await listResourcePdfs(tx, resource.id); + if (now.some((pdf) => pdf.sourceKey === key)) return null; + + await releaseStaleManualArchives(tx, resource.id, key); + const created = await createAttachment({ ...file, sourceKey: key }, { db: tx }); + await claimAttachments(tx, [created.id], { ownerType: "resource", ownerId: resource.id }); + return created.id; + }); + } catch (error) { + // The same race, lost at the unique index instead of the read above. + if (!isUniqueViolation(error)) throw error; + attachmentId = null; + } + if (!attachmentId) { + // The losing run's blob is recorded *unowned*, so the daily sweep deletes + // it after 24 hours instead of it sitting in Blob with no row forever. + await createAttachment(file, { db }).catch(() => {}); + return skip("already_archived", host); + } + + return { + outcome: { + status: "archived", + reason: "archived", + attachmentId, + publicUrl: stored.url, + sizeBytes: downloaded.bytes.byteLength, + }, + host, + }; +} + +// ── The download ──────────────────────────────────────────────────── + +type Download = + | { ok: true; bytes: Uint8Array; filename: string | null } + | { + ok: false; + reason: ArchiveFailReason; + failure: { reason: ArchiveFailReason; transient: boolean; httpStatus?: number }; + }; + +async function download(url: string, options: ArchiveManualOptions): Promise { + const fetchImpl = options.fetchImpl ?? fetch; + const timeout = AbortSignal.timeout(MANUAL_FETCH_TIMEOUT_MS); + const signal = options.signal ? AbortSignal.any([options.signal, timeout]) : timeout; + + let response: Response; + try { + response = await fetchImpl(url, { + headers: { "User-Agent": FETCH_USER_AGENT, Accept: "application/pdf,*/*;q=0.5" }, + redirect: "follow", + signal, + }); + } catch { + return refused("download_failed", true); + } + + if (!response.ok) { + discard(response); + const status = response.status; + return refused("http_error", status >= 500 || status === 429 || status === 408, status); + } + + const declared = Number(response.headers.get("content-length")); + if (Number.isFinite(declared) && declared > MAX_MANUAL_BYTES) { + discard(response); + return refused("too_large", false); + } + + const contentType = response.headers.get("content-type")?.split(";")[0].trim().toLowerCase() ?? ""; + // Markup is refused on the header alone, before a product page's body is + // read — unless it is lying, which the magic bytes would show, and a server + // that labels a PDF `text/html` is not one worth downloading 25 MB to catch. + if (contentType === "text/html" || contentType === "application/xhtml+xml") { + discard(response); + return refused("not_pdf", false); + } + + let bytes: Uint8Array; + try { + const read = await readCapped(response, MAX_MANUAL_BYTES); + if (read === "too_large") return refused("too_large", false); + bytes = read; + } catch { + return refused("download_failed", true); + } + + if (bytes.byteLength === 0) return refused("empty", false); + if (!looksLikePdf(bytes, contentType)) return refused("not_pdf", false); + + return { ok: true, bytes, filename: filenameFromDisposition(response.headers.get("content-disposition")) }; +} + +function refused(reason: ArchiveFailReason, transient: boolean, httpStatus?: number): Download { + return { + ok: false, + reason, + failure: httpStatus === undefined ? { reason, transient } : { reason, transient, httpStatus }, + }; +} + +/** The body, or `"too_large"` the moment it passes `cap` — never more than that is held. */ +async function readCapped(response: Response, cap: number): Promise { + if (!response.body) return new Uint8Array(await response.arrayBuffer()); + const reader = response.body.getReader(); + const chunks: Uint8Array[] = []; + let total = 0; + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + total += value.byteLength; + if (total > cap) { + await reader.cancel().catch(() => {}); + return "too_large"; + } + chunks.push(value); + } + const bytes = new Uint8Array(total); + let offset = 0; + for (const chunk of chunks) { + bytes.set(chunk, offset); + offset += chunk.byteLength; + } + return bytes; +} + +/** + * Let go of a body that is not wanted. Not awaited: a cancel can wait on the + * other end, and nothing here depends on it having finished. + */ +function discard(response: Response): void { + try { + response.body?.cancel().catch(() => {}); + } catch { + // Nothing to do: the body is being thrown away either way. + } +} + +const PDF_MAGIC = [0x25, 0x50, 0x44, 0x46, 0x2d]; // %PDF- + +/** + * `%PDF-` near the start, or a body declared `application/pdf` that does not + * open like markup (some servers gzip-wrap or prefix; an error page still + * starts with `<`). + */ +export function looksLikePdf(bytes: Uint8Array, contentType: string): boolean { + const window = Math.min(bytes.byteLength, PDF_MAGIC_WINDOW); + outer: for (let i = 0; i + PDF_MAGIC.length <= window; i += 1) { + for (let j = 0; j < PDF_MAGIC.length; j += 1) { + if (bytes[i + j] !== PDF_MAGIC[j]) continue outer; + } + return true; + } + if (contentType !== "application/pdf") return false; + const head = new TextDecoder().decode(bytes.subarray(0, 64)).trimStart(); + return !head.startsWith("<"); +} + +/** The name a `Content-Disposition` header gives, if any — display only. */ +export function filenameFromDisposition(header: string | null): string | null { + if (!header) return null; + const extended = /filename\*\s*=\s*(?:UTF-8|utf-8)?''([^;]+)/.exec(header); + if (extended) { + try { + return cleanName(decodeURIComponent(extended[1].trim().replace(/^"|"$/g, ""))); + } catch { + // Malformed percent-encoding: fall through to the plain parameter. + } + } + const plain = /filename\s*=\s*("([^"]*)"|[^;]+)/.exec(header); + if (!plain) return null; + return cleanName((plain[2] ?? plain[1]).trim()); +} + +/** The URL path's last segment, when it names a file. */ +export function filenameFromUrl(url: string): string | null { + try { + const last = new URL(url).pathname.split("/").filter(Boolean).pop(); + return last ? cleanName(decodeURIComponent(last)) : null; + } catch { + return null; + } +} + +function cleanName(name: string): string | null { + const base = name.split(/[\\/]/).pop()?.trim() ?? ""; + return base ? base.slice(0, 200) : null; +} + +// ── Outcomes ──────────────────────────────────────────────────────── + +function skip(reason: ArchiveSkipReason, host?: string): Attempt { + return { outcome: { status: "skipped", reason }, host }; +} + +function fail(reason: ArchiveFailReason, transient: boolean, host?: string): Attempt { + return { outcome: { status: "failed", reason, transient }, host }; +} + +/** Postgres `23505`, read by shape through drizzle's wrapping. */ +function isUniqueViolation(error: unknown): boolean { + let current: unknown = error; + for (let depth = 0; depth < 4 && typeof current === "object" && current !== null; depth += 1) { + if ((current as { code?: unknown }).code === "23505") return true; + current = (current as { cause?: unknown }).cause; + } + return false; +} + +function hostOf(url: string): string | undefined { + try { + return new URL(url).hostname; + } catch { + return undefined; + } +} + +/** One line: the resource, the outcome, the host. Never the path, query or any credential. */ +function logOutcome(resourceId: string, outcome: ArchiveManualResult, host: string | undefined): void { + const where = host ? ` host=${host}` : ""; + if (outcome.status === "failed") { + const status = outcome.httpStatus ? ` status=${outcome.httpStatus}` : ""; + console.warn(`[manuals] archive failed: resource=${resourceId} reason=${outcome.reason}${status}${where}`); + } else if (outcome.status === "archived") { + console.info(`[manuals] archived: resource=${resourceId} bytes=${outcome.sizeBytes}${where}`); + } +} diff --git a/v5/src/lib/manuals/start.ts b/v5/src/lib/manuals/start.ts new file mode 100644 index 0000000..e7916c3 --- /dev/null +++ b/v5/src/lib/manuals/start.ts @@ -0,0 +1,22 @@ +import { start } from "workflow/api"; +import { archiveManuals } from "../../workflows/archive-manuals.ts"; + +/** + * Starting `archiveManuals` — the one module that imports `workflow/api` for + * the manual archive. Callers reach it through `trigger.ts`'s dynamic + * `import()`, so a write with no resources to archive never loads the + * workflow runtime. + * + * True when the run was started. Starting is not archiving: what each + * resource came to is in the run's result and its log line. + */ +export async function startManualArchive(resourceIds: readonly string[]): Promise { + try { + await start(archiveManuals, [[...resourceIds]]); + return true; + } catch (error) { + const name = error instanceof Error ? error.name : "unknown error"; + console.error(`[manuals] could not start an archive run for ${resourceIds.length} resource(s): ${name}`); + return false; + } +} diff --git a/v5/src/lib/manuals/steps.test.ts b/v5/src/lib/manuals/steps.test.ts new file mode 100644 index 0000000..a5ba949 --- /dev/null +++ b/v5/src/lib/manuals/steps.test.ts @@ -0,0 +1,49 @@ +// @vitest-environment node + +/** + * The archive step's retry rule: only a transient failure (network, 5xx, a + * Blob write) or an unreachable database is retried; an HTML page, an + * oversize file or a 404 comes back as a value and is not. + */ + +const archive = vi.hoisted(() => ({ archiveManual: vi.fn() })); +vi.mock("./archive", () => archive); + +import { FatalError, RetryableError } from "workflow"; +import { archiveManualStep, MANUAL_STEP_MAX_RETRIES } from "./steps"; + +const ID = "675596a3-081a-41a5-88e2-91353a18f759"; + +beforeEach(() => { + archive.archiveManual.mockReset(); +}); + +describe("archiveManualStep", () => { + it("returns an archived or skipped outcome as it came", async () => { + archive.archiveManual.mockResolvedValue({ status: "skipped", reason: "already_archived" }); + expect(await archiveManualStep(ID)).toEqual({ status: "skipped", reason: "already_archived" }); + }); + + it("returns a permanent failure without retrying it", async () => { + archive.archiveManual.mockResolvedValue({ status: "failed", reason: "not_pdf", transient: false }); + expect(await archiveManualStep(ID)).toEqual({ status: "failed", reason: "not_pdf", transient: false }); + }); + + it("throws a RetryableError for a transient failure", async () => { + archive.archiveManual.mockResolvedValue({ status: "failed", reason: "http_error", transient: true, httpStatus: 503 }); + const error = await archiveManualStep(ID).catch((e: unknown) => e); + expect(RetryableError.is(error)).toBe(true); + }); + + it("retries an unreachable database and gives up on anything else", async () => { + archive.archiveManual.mockRejectedValueOnce(Object.assign(new Error("x"), { code: "ECONNREFUSED" })); + expect(RetryableError.is(await archiveManualStep(ID).catch((e: unknown) => e))).toBe(true); + + archive.archiveManual.mockRejectedValueOnce(new Error("syntax error at or near")); + expect(FatalError.is(await archiveManualStep(ID).catch((e: unknown) => e))).toBe(true); + }); + + it("sets maxRetries as a property on the step", () => { + expect((archiveManualStep as unknown as { maxRetries: number }).maxRetries).toBe(MANUAL_STEP_MAX_RETRIES); + }); +}); diff --git a/v5/src/lib/manuals/steps.ts b/v5/src/lib/manuals/steps.ts new file mode 100644 index 0000000..baec9d5 --- /dev/null +++ b/v5/src/lib/manuals/steps.ts @@ -0,0 +1,53 @@ +import { FatalError, RetryableError } from "workflow"; +import { isTransientDbError } from "../mirror/steps.ts"; +import { archiveManual, type ArchiveManualResult } from "./archive.ts"; + +/** + * The manual archive's workflow step (see `archive.ts`, and + * `src/workflows/archive-manuals.ts` for the workflow). + * + * **Retries are for the network's bad minute and the database's, nothing + * else.** `archiveManual` answers every expected failure as a value; the ones + * a retry could fix — no answer, a timeout, a 5xx or 429, a Blob write that + * failed — carry `transient: true`, and only those are thrown here as a + * {@link RetryableError}. An HTML product page, an oversize file or a 404 + * returns as a `failed` outcome and is not retried: it would give the same + * answer every time. A throw from `archiveManual` itself is the database — + * retried when it was unreachable, {@link FatalError} otherwise. + * + * `maxRetries` is set **as a property on the step function**, which is how the + * Workflow SDK reads it. Plain Node: relative imports, no `"server-only"`. + */ + +/** Attempts after the first, per resource. */ +export const MANUAL_STEP_MAX_RETRIES = 2; + +const RETRY_AFTER = "1m"; + +export async function archiveManualStep(resourceId: string): Promise { + "use step"; + let result: ArchiveManualResult; + try { + result = await archiveManual(resourceId); + } catch (error) { + if (isTransientDbError(error)) { + throw new RetryableError("Manual archive: the database could not be reached.", { retryAfter: RETRY_AFTER }); + } + throw new FatalError(`Manual archive failed for resource ${resourceId}.`); + } + if (result.status === "failed" && result.transient) { + throw new RetryableError(`Manual archive: ${result.reason} for resource ${resourceId}.`, { + retryAfter: RETRY_AFTER, + }); + } + return result; +} +archiveManualStep.maxRetries = MANUAL_STEP_MAX_RETRIES; + +/** The run is done: one line of counts. */ +export async function finishManualArchive(counts: { archived: number; skipped: number; failed: number }): Promise { + "use step"; + console.info( + `[manuals] archive run finished: archived=${counts.archived} skipped=${counts.skipped} failed=${counts.failed}` + ); +} diff --git a/v5/src/lib/manuals/trigger.test.ts b/v5/src/lib/manuals/trigger.test.ts new file mode 100644 index 0000000..b7f7bbf --- /dev/null +++ b/v5/src/lib/manuals/trigger.test.ts @@ -0,0 +1,33 @@ +// @vitest-environment node + +/** `requestManualArchive` never throws and starts nothing for nothing. */ + +const starter = vi.hoisted(() => ({ startManualArchive: vi.fn() })); +vi.mock("./start", () => starter); + +import { requestManualArchive } from "./trigger"; + +const A = "675596a3-081a-41a5-88e2-91353a18f759"; +const B = "0f5e4a3c-1111-2222-3333-444455556666"; + +beforeEach(() => { + starter.startManualArchive.mockReset().mockResolvedValue(true); + vi.spyOn(console, "error").mockImplementation(() => {}); +}); + +describe("requestManualArchive", () => { + it("starts one run with the distinct, uuid-shaped ids", async () => { + expect(await requestManualArchive([A, B, A, "nope"])).toBe(true); + expect(starter.startManualArchive).toHaveBeenCalledWith([A, B]); + }); + + it("starts nothing for an empty list", async () => { + expect(await requestManualArchive([])).toBe(false); + expect(starter.startManualArchive).not.toHaveBeenCalled(); + }); + + it("answers false, never a throw, when the start blows up", async () => { + starter.startManualArchive.mockRejectedValue(new Error("world unavailable")); + await expect(requestManualArchive([A])).resolves.toBe(false); + }); +}); diff --git a/v5/src/lib/manuals/trigger.ts b/v5/src/lib/manuals/trigger.ts new file mode 100644 index 0000000..067ddf7 --- /dev/null +++ b/v5/src/lib/manuals/trigger.ts @@ -0,0 +1,28 @@ +import { isUuid } from "../data/uuid.ts"; + +/** + * `requestManualArchive(resourceIds)` — "these resources may have a manual + * worth keeping a copy of". + * + * Called after a committed write: approving a tool (`intake/approve.ts`), + * `create_tool` over MCP (`capabilities/intake.ts`), and the tool editor's + * add-resource and edit-resource actions. The daily cron's backfill catches + * anything these miss. + * + * **It never throws, and it never fails the write that called it** (Article + * 4): the resource exists either way, and a copy that could not be started is + * tomorrow night's backfill. Every failure leaves one log line and `false`. + * Nothing is loaded for an empty list; otherwise `start.ts` comes in by a + * dynamic `import()` so `workflow/api` stays out of the callers' static graph. + */ +export async function requestManualArchive(resourceIds: readonly string[]): Promise { + const ids = [...new Set(resourceIds.filter(isUuid))]; + if (ids.length === 0) return false; + try { + const { startManualArchive } = await import("./start.ts"); + return await startManualArchive(ids); + } catch { + console.error("[manuals] could not request a manual archive; the nightly backfill will retry"); + return false; + } +} diff --git a/v5/src/lib/mirror/connect.test.ts b/v5/src/lib/mirror/connect.test.ts new file mode 100644 index 0000000..5a73923 --- /dev/null +++ b/v5/src/lib/mirror/connect.test.ts @@ -0,0 +1,186 @@ +// @vitest-environment node +import { eq } from "drizzle-orm"; +import { server } from "../../../test/msw/server"; +// Aliased: the name starts with `use`, which eslint's rules-of-hooks reads as a React hook. +import { useNotionFake as installNotionFake } from "../../../test/msw/notion-mirror"; +import { createNotionFake, type NotionFake } from "../../../test/fakes/notion-fake"; +import { createPgliteDb } from "../db/pglite"; +import { notionMirrors, user } from "../db/schema/index"; +import type { Db } from "../db/types"; +import { connectMirror, testMirrorConnection } from "./connect"; +import { decryptMirrorToken } from "./token-crypto"; + +/** + * Connecting a mirror against the fake Notion (spec §3.8 "Connect", §8). + * + * Two promises are under test. A token is validated by one read before it is + * stored — so every failure leaves `notion_mirrors` exactly as it was. And the + * token goes nowhere it was not sent: not into a result, not into an error, + * not into a console line. + */ + +const TOKEN = "ntn_CONNECTtestToken0123456789abcdef"; +const SECRET = "connect-test-auth-secret"; +const PAGE_ID = "0f5e4a3c-1111-2222-3333-44445555aaaa"; +const PAGE_URL = `https://www.notion.so/acme/MakerLab-Tools-mirror-${PAGE_ID.replace(/-/g, "")}?pvs=4`; +const TITLE = "MakerLab Tools — mirror"; + +let db: Db; +let fake: NotionFake; +let consoleLines: string[]; + +beforeAll(async () => { + db = await createPgliteDb(); +}); + +beforeEach(() => { + vi.stubEnv("AUTH_SECRET", SECRET); + fake = createNotionFake({ token: TOKEN, pages: [{ id: PAGE_ID, title: TITLE }] }); + installNotionFake(server, fake); + + consoleLines = []; + for (const method of ["log", "info", "warn", "error", "debug"] as const) { + vi.spyOn(console, method).mockImplementation((...args: unknown[]) => { + consoleLines.push(args.map((arg) => (arg instanceof Error ? `${arg.message} ${arg.stack}` : String(arg))).join(" ")); + }); + } +}); + +afterEach(() => { + vi.restoreAllMocks(); + vi.unstubAllEnvs(); + // The token never reaches the console, whatever the test did. + expect(consoleLines.join("\n")).not.toContain(TOKEN); +}); + +async function insertUser(): Promise { + const id = `u-${Math.random().toString(36).slice(2)}`; + await db.insert(user).values({ id, name: "Mirror Owner", email: `${id}@cornell.edu`, role: "admin" }); + return id; +} + +async function mirrorRows(owner: string) { + return db.select().from(notionMirrors).where(eq(notionMirrors.ownerUserId, owner)); +} + +describe("testMirrorConnection", () => { + it("reads the page and answers its id and title", async () => { + const result = await testMirrorConnection(TOKEN, PAGE_URL); + expect(result).toEqual({ ok: true, pageId: PAGE_ID, title: TITLE }); + expect(fake.requests.map((request) => `${request.method} ${request.path}`)).toEqual([ + `GET /pages/${PAGE_ID}`, + ]); + }); + + it("accepts a bare id and a pasted token with stray whitespace around it", async () => { + expect(await testMirrorConnection(` ${TOKEN}\n`, PAGE_ID.replace(/-/g, ""))).toMatchObject({ ok: true }); + }); + + it("refuses a token that is not token-shaped without calling Notion", async () => { + for (const bad of ["", "short", "ntn_has a space inside it 0123", "x".repeat(201), 42, null]) { + expect(await testMirrorConnection(bad, PAGE_URL)).toEqual({ ok: false, code: "invalid_token" }); + } + expect(fake.requests).toHaveLength(0); + }); + + it("refuses a page reference with no Notion id in it", async () => { + for (const bad of ["", "https://example.com/page", "not a url", "https://www.notion.so/acme/No-id-here"]) { + expect(await testMirrorConnection(TOKEN, bad)).toEqual({ ok: false, code: "invalid_page" }); + } + expect(fake.requests).toHaveLength(0); + }); + + it("maps a wrong token to unauthorized", async () => { + expect(await testMirrorConnection(`${TOKEN}WRONG`, PAGE_URL)).toEqual({ ok: false, code: "unauthorized" }); + }); + + it("maps an unshared or missing page to page_not_found", async () => { + expect(await testMirrorConnection(TOKEN, crypto.randomUUID())).toEqual({ ok: false, code: "page_not_found" }); + + fake.failNext({ method: "GET" }, { status: 403, code: "restricted_resource" }); + expect(await testMirrorConnection(TOKEN, PAGE_URL)).toEqual({ ok: false, code: "page_not_found" }); + }); + + it("maps Notion being down to notion_unavailable", async () => { + fake.failNext({ method: "GET" }, { status: 503, code: "service_unavailable" }); + expect(await testMirrorConnection(TOKEN, PAGE_URL)).toEqual({ ok: false, code: "notion_unavailable" }); + }); + + it("never carries the token in a result", async () => { + const results = [ + await testMirrorConnection(TOKEN, PAGE_URL), + await testMirrorConnection(TOKEN, crypto.randomUUID()), + await testMirrorConnection(`${TOKEN}WRONG`, PAGE_URL), + ]; + expect(JSON.stringify(results)).not.toContain(TOKEN); + }); +}); + +describe("connectMirror", () => { + it("stores the page, its title and the token encrypted — never in plain text", async () => { + const owner = await insertUser(); + const result = await connectMirror(owner, TOKEN, PAGE_URL, { db }); + + expect(result).toMatchObject({ ok: true, created: true, pageId: PAGE_ID, title: TITLE }); + expect(JSON.stringify(result)).not.toContain(TOKEN); + + const [row] = await mirrorRows(owner); + expect(row.parentPageId).toBe(PAGE_ID); + expect(row.parentPageTitle).toBe(TITLE); + expect(row.tokenCiphertext).not.toBeNull(); + expect(Buffer.from(row.tokenCiphertext as Uint8Array).toString("utf8")).not.toContain(TOKEN); + expect(decryptMirrorToken(row.tokenCiphertext as Uint8Array)).toBe(TOKEN); + }); + + it("reconnects in place, keeping the mirror's id", async () => { + const owner = await insertUser(); + const first = await connectMirror(owner, TOKEN, PAGE_URL, { db }); + const second = await connectMirror(owner, TOKEN, PAGE_ID, { db }); + expect(second).toMatchObject({ ok: true, created: false }); + if (!first.ok || !second.ok) throw new Error("expected both to connect"); + expect(second.mirror.id).toBe(first.mirror.id); + expect(await mirrorRows(owner)).toHaveLength(1); + }); + + it("stores nothing when the read fails, whatever the reason", async () => { + const owner = await insertUser(); + + expect(await connectMirror(owner, "short", PAGE_URL, { db })).toEqual({ ok: false, code: "invalid_token" }); + expect(await connectMirror(owner, TOKEN, "https://example.com", { db })).toEqual({ + ok: false, + code: "invalid_page", + }); + expect(await connectMirror(owner, `${TOKEN}WRONG`, PAGE_URL, { db })).toEqual({ + ok: false, + code: "unauthorized", + }); + expect(await connectMirror(owner, TOKEN, crypto.randomUUID(), { db })).toEqual({ + ok: false, + code: "page_not_found", + }); + fake.failNext({ method: "GET" }, { status: 500 }); + expect(await connectMirror(owner, TOKEN, PAGE_URL, { db })).toEqual({ ok: false, code: "notion_unavailable" }); + + expect(await mirrorRows(owner)).toHaveLength(0); + }); + + it("does not replace a working token with one that fails its read", async () => { + const owner = await insertUser(); + await connectMirror(owner, TOKEN, PAGE_URL, { db }); + const [before] = await mirrorRows(owner); + + expect(await connectMirror(owner, `${TOKEN}WRONG`, PAGE_URL, { db })).toEqual({ + ok: false, + code: "unauthorized", + }); + const [after] = await mirrorRows(owner); + expect(Buffer.from(after.tokenCiphertext as Uint8Array)).toEqual(Buffer.from(before.tokenCiphertext as Uint8Array)); + }); + + it("refuses with key_unavailable, and stores nothing, when AUTH_SECRET is unset", async () => { + vi.stubEnv("AUTH_SECRET", ""); + const owner = await insertUser(); + expect(await connectMirror(owner, TOKEN, PAGE_URL, { db })).toEqual({ ok: false, code: "key_unavailable" }); + expect(await mirrorRows(owner)).toHaveLength(0); + }); +}); diff --git a/v5/src/lib/mirror/connect.ts b/v5/src/lib/mirror/connect.ts new file mode 100644 index 0000000..1360d62 --- /dev/null +++ b/v5/src/lib/mirror/connect.ts @@ -0,0 +1,153 @@ +import { z } from "zod"; +import { saveMirrorConnection, type MirrorRecord } from "../data/mirrors.ts"; +import type { Db } from "../db/types.ts"; +import { createNotionClient, NotionMirrorError, pageTitle, type NotionClientOptions } from "./notion-client.ts"; +import { parseNotionId } from "./notion-id.ts"; +import { MirrorKeyUnavailableError, encryptMirrorToken, mirrorKeyAvailable } from "./token-crypto.ts"; +import type { MirrorSetupError } from "./types.ts"; + +/** + * Connecting a mirror (spec §3.8 "Connect", §4.14, §8). + * + * **A token is validated by one read before it is stored, never after** (§8 + * "Untrusted input"). `testMirrorConnection` reads the shared page with the + * pasted token; `connectMirror` runs that same read and stores nothing unless + * it succeeded. So a mistyped token, or a page nobody shared with the + * integration, is a sentence on the page and not a row that fails every push + * from now on. + * + * **The token goes three places and no fourth**: the `Authorization` header of + * that one read, `encryptMirrorToken`, and nowhere else. Every refusal is a + * code, never a message built from what was typed, and nothing here logs — + * `NotionMirrorError`'s own text is scrubbed of the token, but the safest line + * is the one that is not written. + * + * Not step code, but relative imports with `.ts` extensions like the rest of + * `src/lib/mirror/`. + */ + +/** How long Test connection may take. A settings page that hangs is a broken settings page. */ +export const MIRROR_TEST_TIMEOUT_MS = 10_000; + +/** + * An internal-integration token as Notion issues them (`ntn_…`, and the older + * `secret_…`): one unbroken run of characters. The bounds are generous; what + * they reject is an empty box, a paragraph, and a paste that caught a space. + */ +export const mirrorTokenSchema = z + .string() + .trim() + .min(20) + .max(200) + .regex(/^\S+$/); + +/** A page URL or id, as a browser's address bar shows it. Bounded before it is parsed. */ +const pageRefSchema = z.string().trim().min(1).max(2048); + +export interface MirrorConnectOptions { + db?: Db; + /** Test seams for the one Notion read (base URL, clock). Never the token. */ + client?: Partial>; +} + +export type MirrorTestConnectionResult = + | { ok: true; pageId: string; title: string | null } + | { ok: false; code: MirrorSetupError }; + +export type MirrorConnectResult = + | { ok: true; mirror: MirrorRecord; created: boolean; pageId: string; title: string | null } + | { ok: false; code: MirrorSetupError }; + +/** + * **Test connection**: read the page with the pasted token and say its title. + * Stores nothing. + * + * - a token that is not token-shaped → `invalid_token`, and no request is made; + * - a page reference with no Notion id in it → `invalid_page`; + * - 401 → `unauthorized` (the token is wrong or was revoked); + * - 404, or 403 (the page exists but was not shared with the integration), or + * a page in the trash → `page_not_found`: the remedy is the same, share the + * page with the integration; + * - anything else — 5xx, a timeout, a rate limit → `notion_unavailable`. + */ +export async function testMirrorConnection( + token: unknown, + pageRef: unknown, + options: MirrorConnectOptions = {} +): Promise { + const parsedToken = mirrorTokenSchema.safeParse(token); + if (!parsedToken.success) return { ok: false, code: "invalid_token" }; + + const parsedRef = pageRefSchema.safeParse(pageRef); + const pageId = parsedRef.success ? parseNotionId(parsedRef.data) : null; + if (!pageId) return { ok: false, code: "invalid_page" }; + + const now = options.client?.now ?? Date.now; + const client = createNotionClient({ + ...options.client, + token: parsedToken.data, + deadline: now() + MIRROR_TEST_TIMEOUT_MS, + }); + + try { + const page = await client.getPage(pageId); + if (page.archived || page.in_trash) return { ok: false, code: "page_not_found" }; + return { ok: true, pageId, title: pageTitle(page) }; + } catch (error) { + return { ok: false, code: setupCodeFor(error) }; + } +} + +/** + * **Connect**: the same read, then — only if it succeeded — encrypt the token + * and upsert the owner's mirror with the page's id and title. + * + * Reconnecting keeps the mapping and every `mirror_pages` row + * (`saveMirrorConnection`), so the same Notion pages are updated rather than + * duplicated, and clears a pause or an error the old token caused (§5.8). + * + * `key_unavailable` when `AUTH_SECRET` is unset: the token checked out, and + * there is no key to keep it under, so it is not kept. + */ +export async function connectMirror( + ownerUserId: string, + token: unknown, + pageRef: unknown, + options: MirrorConnectOptions = {} +): Promise { + const tested = await testMirrorConnection(token, pageRef, options); + if (!tested.ok) return tested; + + if (!mirrorKeyAvailable()) return { ok: false, code: "key_unavailable" }; + + let tokenCiphertext: Uint8Array; + try { + // Parsed again rather than threaded through: the test result deliberately + // does not carry the token, so no caller can hand it onward by accident. + tokenCiphertext = encryptMirrorToken(mirrorTokenSchema.parse(token)); + } catch (error) { + if (error instanceof MirrorKeyUnavailableError) return { ok: false, code: "key_unavailable" }; + throw new Error("The mirror token could not be encrypted."); + } + + const { mirror, created } = await saveMirrorConnection( + { ownerUserId, tokenCiphertext, parentPageId: tested.pageId, parentPageTitle: tested.title }, + { db: options.db } + ); + return { ok: true, mirror, created, pageId: tested.pageId, title: tested.title }; +} + +/** What a failed page read means to the person holding the token. */ +function setupCodeFor(error: unknown): MirrorSetupError { + if (!(error instanceof NotionMirrorError)) return "notion_unavailable"; + switch (error.code) { + case "unauthorized": + return "unauthorized"; + case "page_not_found": + case "not_found": + case "restricted": + return "page_not_found"; + default: + return "notion_unavailable"; + } +} diff --git a/v5/src/lib/mirror/credentials.test.ts b/v5/src/lib/mirror/credentials.test.ts new file mode 100644 index 0000000..78c5fee --- /dev/null +++ b/v5/src/lib/mirror/credentials.test.ts @@ -0,0 +1,49 @@ +// @vitest-environment node +import { http, HttpResponse } from "msw"; +import { server } from "../../../test/msw/server"; +import { mirrorClientFor } from "./credentials"; +import { encryptMirrorToken } from "./token-crypto"; + +const TOKEN = "ntn_TESTtoken0123456789abcdefABCDEF"; +const SECRET = "test-auth-secret-for-mirror-credentials"; +const PAGE_ID = "0f5e4a3c-1111-2222-3333-44445555aaaa"; + +describe("mirrorClientFor", () => { + it("is not_connected with no stored token", () => { + vi.stubEnv("AUTH_SECRET", SECRET); + expect(mirrorClientFor(null)).toEqual({ ok: false, code: "not_connected" }); + expect(mirrorClientFor(new Uint8Array())).toEqual({ ok: false, code: "not_connected" }); + }); + + it("is key_unavailable without AUTH_SECRET", () => { + const stored = encryptMirrorToken(TOKEN, SECRET); + vi.stubEnv("AUTH_SECRET", ""); + expect(mirrorClientFor(stored)).toEqual({ ok: false, code: "key_unavailable" }); + }); + + it("is token_unreadable after AUTH_SECRET rotates", () => { + const stored = encryptMirrorToken(TOKEN, SECRET); + vi.stubEnv("AUTH_SECRET", "a-rotated-auth-secret"); + expect(mirrorClientFor(stored)).toEqual({ ok: false, code: "token_unreadable" }); + }); + + it("builds a client that sends the decrypted token and honours the options", async () => { + vi.stubEnv("AUTH_SECRET", SECRET); + let authorization: string | null = null; + server.use( + http.get("https://api.notion.com/v1/pages/:id", ({ request, params }) => { + authorization = request.headers.get("authorization"); + return HttpResponse.json({ object: "page", id: params.id, properties: {} }); + }) + ); + + const result = mirrorClientFor(encryptMirrorToken(TOKEN), { deadline: 42, now: () => 0 }); + expect(result.ok).toBe(true); + if (!result.ok) return; + expect(result.client.remainingMs()).toBe(42); + await result.client.getPage(PAGE_ID); + expect(authorization).toBe(`Bearer ${TOKEN}`); + // The result carries a client, never the plaintext. + expect(JSON.stringify(result)).not.toContain(TOKEN); + }); +}); diff --git a/v5/src/lib/mirror/credentials.ts b/v5/src/lib/mirror/credentials.ts new file mode 100644 index 0000000..a3a6875 --- /dev/null +++ b/v5/src/lib/mirror/credentials.ts @@ -0,0 +1,42 @@ +import { createNotionClient, type NotionClient, type NotionClientOptions } from "./notion-client.ts"; +import { + MirrorKeyUnavailableError, + MirrorTokenUnreadableError, + decryptMirrorToken, +} from "./token-crypto.ts"; + +/** + * A Notion client for a stored mirror, or the reason there cannot be one + * (spec §8 "Secrets at rest", §5.8). + * + * The one place a stored token is decrypted. The plaintext goes straight into + * the client and is returned nowhere else, so no caller can log it by + * accident. The three refusals are values, not exceptions, because each is a + * state the mirror page explains rather than a crash: + * + * - `not_connected` — no token is stored (never connected, or disconnected); + * - `key_unavailable` — `AUTH_SECRET` is not set, so no key can be derived; + * - `token_unreadable` — the stored value does not decrypt under the current + * key, which is what rotating `AUTH_SECRET` does. Connect again. + */ +export type MirrorClientResult = + | { ok: true; client: NotionClient } + | { ok: false; code: "not_connected" | "key_unavailable" | "token_unreadable" }; + +export function mirrorClientFor( + tokenCiphertext: Uint8Array | null, + options: Omit = {} +): MirrorClientResult { + if (!tokenCiphertext || tokenCiphertext.length === 0) return { ok: false, code: "not_connected" }; + let token: string; + try { + token = decryptMirrorToken(tokenCiphertext); + } catch (error) { + if (error instanceof MirrorKeyUnavailableError) return { ok: false, code: "key_unavailable" }; + if (error instanceof MirrorTokenUnreadableError) return { ok: false, code: "token_unreadable" }; + // decryptMirrorToken throws nothing else; treat anything that slips + // through as unreadable rather than letting it carry a message upward. + return { ok: false, code: "token_unreadable" }; + } + return { ok: true, client: createNotionClient({ ...options, token }) }; +} diff --git a/v5/src/lib/mirror/database-schemas.test.ts b/v5/src/lib/mirror/database-schemas.test.ts new file mode 100644 index 0000000..80c9ed2 --- /dev/null +++ b/v5/src/lib/mirror/database-schemas.test.ts @@ -0,0 +1,114 @@ +// @vitest-environment node +import { MIRROR_ENTITY } from "../db/schema/vocabulary"; +import { + databaseCreateBody, + expectedProperties, + mirrorDatabaseTitle, + mirrorPropertySpecs, + validateDatabaseSchema, +} from "./database-schemas"; +import type { NotionDatabaseObject } from "./notion-client"; + +/** The fixed Notion schemas and their validation (spec §3.8 "Mapping"). */ + +const CATEGORIES_DB = "aaaaaaaa-0000-4000-8000-000000000001"; +const LOCATIONS_DB = "aaaaaaaa-0000-4000-8000-000000000002"; +const OTHER_DB = "aaaaaaaa-0000-4000-8000-0000000000ff"; + +/** A database object as Notion returns it for the tools schema under `mapping`. */ +function toolsDatabase(): NotionDatabaseObject { + const properties: NotionDatabaseObject["properties"] = {}; + for (const expected of expectedProperties("tools", { categories: CATEGORIES_DB, locations: LOCATIONS_DB })) { + properties[expected.name] = { + id: expected.name, + name: expected.name, + type: expected.type, + ...(expected.type === "relation" ? { relation: { database_id: expected.targetDatabaseId ?? undefined } } : {}), + }; + } + return { object: "database", id: OTHER_DB, properties }; +} + +const MAPPING = { categories: CATEGORIES_DB, locations: LOCATIONS_DB }; + +describe("mirror database schemas", () => { + it("gives every database exactly one title, an App ID and an Updated date", () => { + for (const entity of MIRROR_ENTITY) { + const specs = mirrorPropertySpecs(entity); + expect(specs.filter((spec) => spec.type === "title"), entity).toHaveLength(1); + expect(specs.find((spec) => spec.name === "App ID")?.type).toBe("rich_text"); + expect(specs.find((spec) => spec.name === "Updated")?.type).toBe("date"); + expect(new Set(specs.map((spec) => spec.name)).size, entity).toBe(specs.length); + } + }); + + it("carries the email properties the 2026-09-23 amendment added", () => { + const maintenance = Object.fromEntries(mirrorPropertySpecs("maintenance").map((spec) => [spec.name, spec.type])); + expect(maintenance["Reporter email"]).toBe("email"); + expect(maintenance["Assignee email"]).toBe("email"); + const projects = Object.fromEntries(mirrorPropertySpecs("projects").map((spec) => [spec.name, spec.type])); + expect(projects["Author email"]).toBe("email"); + }); + + it("builds a create body under the parent page, with one-way relations to the mapped targets", () => { + const body = databaseCreateBody("tools", "page-id", MAPPING) as { + parent: unknown; + title: { text: { content: string } }[]; + description: { text: { content: string } }[]; + properties: Record>; + }; + expect(body.parent).toEqual({ type: "page_id", page_id: "page-id" }); + expect(body.title[0].text.content).toBe(mirrorDatabaseTitle("tools")); + expect(body.title[0].text.content).toBe("MakerLab Tools — Tools"); + expect(body.description[0].text.content).toBe( + "Mirrored one way from MakerLab Tools. Edits made here are overwritten by the next push." + ); + expect(body.properties.Category).toEqual({ + relation: { database_id: CATEGORIES_DB, type: "single_property", single_property: {} }, + }); + expect(body.properties.Name).toEqual({ title: {} }); + expect(body.properties.Published).toEqual({ checkbox: {} }); + + const units = databaseCreateBody("units", "page-id", {}) as { properties: Record> }; + expect("Tool" in units.properties).toBe(false); + expect(units.properties.Status.select.options.map((option) => option.name)).toContain("in_use"); + }); + + it("accepts a database with the expected schema, extra properties allowed", () => { + const database = toolsDatabase(); + database.properties["Somebody's column"] = { type: "number" }; + expect(validateDatabaseSchema("tools", database, MAPPING)).toBeNull(); + }); + + it("reports a missing property", () => { + const database = toolsDatabase(); + delete database.properties["Emergency stop"]; + expect(validateDatabaseSchema("tools", database, MAPPING)).toEqual({ + entity: "tools", + code: "schema_mismatch", + missing: ["Emergency stop"], + }); + }); + + it("reports a property of the wrong type", () => { + const database = toolsDatabase(); + database.properties.Published = { type: "rich_text" }; + expect(validateDatabaseSchema("tools", database, MAPPING)).toEqual({ + entity: "tools", + code: "schema_mismatch", + wrongType: ["Published"], + }); + }); + + it("reports a relation pointing at the wrong database, but only when its target is mapped", () => { + const database = toolsDatabase(); + database.properties.Category = { type: "relation", relation: { database_id: OTHER_DB } }; + expect(validateDatabaseSchema("tools", database, MAPPING)).toMatchObject({ wrongType: ["Category"] }); + // Undashed ids compare equal to dashed ones. + database.properties.Category = { type: "relation", relation: { database_id: CATEGORIES_DB.replace(/-/g, "") } }; + expect(validateDatabaseSchema("tools", database, MAPPING)).toBeNull(); + // With categories unmapped, any relation target is accepted. + database.properties.Category = { type: "relation", relation: { database_id: OTHER_DB } }; + expect(validateDatabaseSchema("tools", database, { locations: LOCATIONS_DB })).toBeNull(); + }); +}); diff --git a/v5/src/lib/mirror/database-schemas.ts b/v5/src/lib/mirror/database-schemas.ts new file mode 100644 index 0000000..1368220 --- /dev/null +++ b/v5/src/lib/mirror/database-schemas.ts @@ -0,0 +1,281 @@ +import { + MIRROR_ENTITY, + MAINTENANCE_PRIORITY, + MAINTENANCE_STATUS, + MAINTENANCE_TYPE, + UNIT_CONDITION, + UNIT_STATUS, + type MirrorEntity, +} from "../db/schema/vocabulary.ts"; +import type { NotionDatabaseObject } from "./notion-client.ts"; +import { parseNotionId } from "./notion-id.ts"; +import type { MappingProblem, MirrorMapping } from "./types.ts"; + +/** + * The fixed Notion schema of each mirror database (spec §3.8 "Mapping"). + * + * **This is a schema contract, not app UI.** The property names are fixed + * English in code, as a column name is: the push writes by name, a pasted + * database is validated by name, and translating them would make two admins' + * mirrors disagree. Nothing here reaches `messages/*.json`. + * + * Every database has an `App ID` (the Postgres uuid, so a Notion row can be + * traced back) and an `Updated` date. Relations are one-way + * (`single_property`) and point at the database the mapping holds for their + * target entity. Select values are the stored machine ids verbatim + * (`in_use`), so a filter written in Notion keeps working when a display + * label changes. + * + * Relative imports with `.ts` extensions, no `server-only`: workflow step code + * loads this module. + */ + +export type MirrorPropertyType = + | "title" + | "rich_text" + | "select" + | "multi_select" + | "checkbox" + | "date" + | "url" + | "email" + | "files" + | "relation"; + +export interface MirrorPropertySpec { + name: string; + type: MirrorPropertyType; + /** For a relation: the entity whose database it points at. */ + target?: MirrorEntity; + /** For a select: the options created with the database (Notion adds any others on first use). */ + options?: readonly string[]; +} + +/** The property every database carries: the Postgres uuid. */ +export const APP_ID_PROPERTY = "App ID"; +/** The property every database carries: the row's `updated_at`. */ +export const UPDATED_PROPERTY = "Updated"; + +export const MIRROR_DATABASE_DESCRIPTION = + "Mirrored one way from MakerLab Tools. Edits made here are overwritten by the next push."; + +/** The entity as a database title word. */ +const ENTITY_TITLES: Readonly> = { + categories: "Categories", + locations: "Locations", + tools: "Tools", + units: "Units", + resources: "Resources", + maintenance: "Maintenance", + projects: "Projects", +}; + +/** "MakerLab Tools — Tools" and so on. */ +export function mirrorDatabaseTitle(entity: MirrorEntity): string { + return `MakerLab Tools — ${ENTITY_TITLES[entity]}`; +} + +const COMMON: readonly MirrorPropertySpec[] = [ + { name: APP_ID_PROPERTY, type: "rich_text" }, + { name: UPDATED_PROPERTY, type: "date" }, +]; + +const SCHEMAS: Readonly> = { + categories: [ + { name: "Name", type: "title" }, + { name: "Group", type: "rich_text" }, + ], + locations: [ + { name: "Name", type: "title" }, + { name: "Room", type: "rich_text" }, + { name: "Zone", type: "rich_text" }, + { name: "Map tag", type: "rich_text" }, + ], + tools: [ + { name: "Name", type: "title" }, + { name: "Slug", type: "rich_text" }, + { name: "Description", type: "rich_text" }, + { name: "Category", type: "relation", target: "categories" }, + { name: "Location", type: "relation", target: "locations" }, + { name: "Materials", type: "multi_select" }, + { name: "PPE required", type: "multi_select" }, + { name: "Tags", type: "multi_select" }, + { name: "Training required", type: "checkbox" }, + { name: "Use restrictions", type: "rich_text" }, + { name: "Emergency stop", type: "rich_text" }, + { name: "Notes", type: "rich_text" }, + { name: "Published", type: "checkbox" }, + { name: "Archived", type: "checkbox" }, + { name: "Images", type: "files" }, + { name: "Last reviewed", type: "date" }, + ], + units: [ + { name: "Label", type: "title" }, + { name: "Tool", type: "relation", target: "tools" }, + { name: "Serial number", type: "rich_text" }, + { name: "Asset tag", type: "rich_text" }, + { name: "Status", type: "select", options: UNIT_STATUS }, + { name: "Condition", type: "select", options: UNIT_CONDITION }, + { name: "Date acquired", type: "date" }, + { name: "Notes", type: "rich_text" }, + ], + resources: [ + { name: "Title", type: "title" }, + { name: "Tool", type: "relation", target: "tools" }, + { name: "Type", type: "select" }, + { name: "URL", type: "url" }, + { name: "File", type: "files" }, + { name: "Published", type: "checkbox" }, + { name: "Notes", type: "rich_text" }, + ], + maintenance: [ + { name: "Title", type: "title" }, + { name: "Type", type: "select", options: MAINTENANCE_TYPE }, + { name: "Priority", type: "select", options: MAINTENANCE_PRIORITY }, + { name: "Status", type: "select", options: MAINTENANCE_STATUS }, + { name: "Description", type: "rich_text" }, + { name: "Resolution", type: "rich_text" }, + { name: "Tool", type: "relation", target: "tools" }, + { name: "Unit", type: "relation", target: "units" }, + { name: "Tool name", type: "rich_text" }, + { name: "Unit label", type: "rich_text" }, + { name: "Reported by", type: "rich_text" }, + { name: "Reporter email", type: "email" }, + { name: "Assigned to", type: "rich_text" }, + { name: "Assignee email", type: "email" }, + { name: "Date reported", type: "date" }, + { name: "Date resolved", type: "date" }, + ], + projects: [ + { name: "Title", type: "title" }, + { name: "Link", type: "url" }, + { name: "Body", type: "rich_text" }, + { name: "Materials", type: "multi_select" }, + { name: "Tools", type: "relation", target: "tools" }, + { name: "Author", type: "rich_text" }, + { name: "Author email", type: "email" }, + { name: "Published at", type: "date" }, + { name: "Photos", type: "files" }, + ], +}; + +/** An expected property, with the database id a relation must point at when its target is mapped. */ +export interface ExpectedProperty extends MirrorPropertySpec { + /** Dashed lower-case database id of the relation's target, or null when the target is not mapped. */ + targetDatabaseId?: string | null; +} + +/** Every property `entity`'s database must have, in declaration order (the common two last). */ +export function mirrorPropertySpecs(entity: MirrorEntity): readonly MirrorPropertySpec[] { + return [...SCHEMAS[entity], ...COMMON]; +} + +/** The expected properties of `entity`'s database under `mapping`. */ +export function expectedProperties(entity: MirrorEntity, mapping: MirrorMapping): ExpectedProperty[] { + return mirrorPropertySpecs(entity).map((spec) => + spec.type === "relation" && spec.target + ? { ...spec, targetDatabaseId: normalizeDatabaseId(mapping[spec.target]) } + : { ...spec } + ); +} + +/** The relation properties of `entity`, with their target entities. */ +export function relationProperties(entity: MirrorEntity): { name: string; target: MirrorEntity }[] { + return SCHEMAS[entity] + .filter((spec) => spec.type === "relation" && spec.target) + .map((spec) => ({ name: spec.name, target: spec.target as MirrorEntity })); +} + +/** + * The entities with a relation property pointing at any of `targets`, not + * counting `targets` themselves — whose pages link to a target's pages. When a + * target's pages are forgotten (a recreated or repointed database), these + * pages carry links into the old database and must be pushed again. + */ +export function relationDependents(targets: readonly MirrorEntity[]): MirrorEntity[] { + const set = new Set(targets); + return MIRROR_ENTITY.filter( + (entity) => !set.has(entity) && relationProperties(entity).some((relation) => set.has(relation.target)) + ); +} + +/** + * Whether `database` has every property the push writes, with the right type + * — and, for a relation whose target is mapped, pointing at that target. + * Extra properties are allowed; an admin may add their own columns. + * + * A relation pointing at the wrong database is reported in `wrongType`: it has + * the right Notion type but the wrong shape for this mirror. + */ +export function validateDatabaseSchema( + entity: MirrorEntity, + database: NotionDatabaseObject, + mapping: MirrorMapping +): MappingProblem | null { + const missing: string[] = []; + const wrongType: string[] = []; + const properties = database.properties ?? {}; + for (const expected of expectedProperties(entity, mapping)) { + const actual = properties[expected.name]; + if (!actual) { + missing.push(expected.name); + continue; + } + if (actual.type !== expected.type) { + wrongType.push(expected.name); + continue; + } + if (expected.type === "relation" && expected.targetDatabaseId) { + const pointsAt = normalizeDatabaseId(actual.relation?.database_id); + if (pointsAt !== expected.targetDatabaseId) wrongType.push(expected.name); + } + } + if (missing.length === 0 && wrongType.length === 0) return null; + return { + entity, + code: "schema_mismatch", + ...(missing.length ? { missing } : {}), + ...(wrongType.length ? { wrongType } : {}), + }; +} + +/** One property's schema as `POST /databases` / `PATCH /databases/:id` take it. */ +export function propertySchemaBody(spec: MirrorPropertySpec, targetDatabaseId: string | null): Record | null { + switch (spec.type) { + case "relation": + if (!targetDatabaseId) return null; + return { relation: { database_id: targetDatabaseId, type: "single_property", single_property: {} } }; + case "select": + return { select: { options: (spec.options ?? []).map((name) => ({ name })) } }; + case "multi_select": + return { multi_select: { options: [] } }; + default: + return { [spec.type]: {} }; + } +} + +/** + * The `POST /databases` body for `entity` under `parentPageId`. A relation + * whose target has no database yet is left out; `ensureMirrorDatabases` + * creates in dependency order, so that only happens to a caller that skipped + * a target. + */ +export function databaseCreateBody(entity: MirrorEntity, parentPageId: string, mapping: MirrorMapping): Record { + const properties: Record = {}; + for (const expected of expectedProperties(entity, mapping)) { + const body = propertySchemaBody(expected, expected.targetDatabaseId ?? null); + if (body) properties[expected.name] = body; + } + return { + parent: { type: "page_id", page_id: parentPageId }, + title: [{ type: "text", text: { content: mirrorDatabaseTitle(entity) } }], + description: [{ type: "text", text: { content: MIRROR_DATABASE_DESCRIPTION } }], + properties, + }; +} + +/** A database id in the form Notion returns and the mapping stores, or null. */ +export function normalizeDatabaseId(value: string | null | undefined): string | null { + if (typeof value !== "string") return null; + return parseNotionId(value); +} diff --git a/v5/src/lib/mirror/databases.test.ts b/v5/src/lib/mirror/databases.test.ts new file mode 100644 index 0000000..dc47b29 --- /dev/null +++ b/v5/src/lib/mirror/databases.test.ts @@ -0,0 +1,255 @@ +// @vitest-environment node +import { and, eq } from "drizzle-orm"; +import { server } from "../../../test/msw/server"; +// Aliased: the name starts with `use`, which eslint's rules-of-hooks reads as a React hook. +import { useNotionFake as installNotionFake } from "../../../test/msw/notion-mirror"; +import { createNotionFake, type NotionFake } from "../../../test/fakes/notion-fake"; +import { createPgliteDb } from "../db/pglite"; +import { mirrorPages, notionMirrors, user } from "../db/schema/index"; +import { MIRROR_ENTITY, type MirrorEntity } from "../db/schema/vocabulary"; +import type { Db } from "../db/types"; +import { upsertMirrorPage } from "../data/mirror-pages"; +import { getMirror, saveMirrorConnection } from "../data/mirrors"; +import { applyPastedMapping, ensureMirrorDatabases } from "./databases"; +import { mirrorDatabaseTitle } from "./database-schemas"; +import { encryptMirrorToken } from "./token-crypto"; + +/** Create databases and pasted mappings against the fake Notion (spec §3.8, §5.8). */ + +const TOKEN = "ntn_DATABASEStoken0123456789abcdef"; +const SECRET = "test-auth-secret-for-mirror-databases"; +const PARENT = "0f5e4a3c-1111-2222-3333-44445555aaaa"; +const CLIENT = { requestsPerSecond: 0 }; + +let db: Db; +let fake: NotionFake; + +beforeAll(async () => { + db = await createPgliteDb(); +}); + +beforeEach(() => { + vi.stubEnv("AUTH_SECRET", SECRET); + fake = createNotionFake({ token: TOKEN, pages: [{ id: PARENT, title: "MakerLab Tools — mirror" }] }); + installNotionFake(server, fake); +}); + +async function connect(token = TOKEN, parent = PARENT): Promise { + const owner = `u-${crypto.randomUUID()}`; + await db.insert(user).values({ id: owner, name: "Owner", email: `${owner}@cornell.edu`, role: "admin" }); + const { mirror } = await saveMirrorConnection( + { ownerUserId: owner, tokenCiphertext: encryptMirrorToken(token, SECRET), parentPageId: parent, parentPageTitle: "Mirror" }, + { db } + ); + return mirror.id; +} + +function titleOf(body: unknown): string { + return ((body as { title?: { text?: { content?: string } }[] }).title ?? [])[0]?.text?.content ?? ""; +} + +async function pagesFor(mirrorId: string, entity: MirrorEntity): Promise { + const rows = await db + .select({ id: mirrorPages.entityId }) + .from(mirrorPages) + .where(and(eq(mirrorPages.mirrorId, mirrorId), eq(mirrorPages.entity, entity))); + return rows.length; +} + +describe("ensureMirrorDatabases", () => { + it("creates all seven under the page, in dependency order, with relations resolved", async () => { + const mirrorId = await connect(); + const result = await ensureMirrorDatabases(mirrorId, { db, client: CLIENT }); + if (!result.ok) throw new Error(`expected ok, got ${result.code}`); + + expect(result.created).toEqual([...MIRROR_ENTITY]); + expect(result.kept).toEqual([]); + const creates = fake.requests.filter((request) => request.method === "POST" && request.path === "/databases"); + expect(creates.map((request) => titleOf(request.body))).toEqual(MIRROR_ENTITY.map(mirrorDatabaseTitle)); + + const mapping = result.mapping; + expect(Object.keys(mapping)).toEqual([...MIRROR_ENTITY]); + const toolsDb = fake.databases.get(mapping.tools!)!; + expect(toolsDb.parent).toEqual({ type: "page_id", page_id: PARENT }); + expect(toolsDb.properties.Category.relation).toMatchObject({ database_id: mapping.categories }); + expect(toolsDb.properties.Location.relation).toMatchObject({ database_id: mapping.locations }); + expect(fake.databases.get(mapping.maintenance!)!.properties.Unit.relation).toMatchObject({ database_id: mapping.units }); + expect(fake.databases.get(mapping.projects!)!.properties.Tools.relation).toMatchObject({ database_id: mapping.tools }); + + expect((await getMirror(mirrorId, { db }))!.mapping).toEqual(mapping); + }); + + it("keeps every database on a second call and creates nothing", async () => { + const mirrorId = await connect(); + const first = await ensureMirrorDatabases(mirrorId, { db, client: CLIENT }); + fake.requests.length = 0; + const second = await ensureMirrorDatabases(mirrorId, { db, client: CLIENT }); + expect(second).toEqual({ ok: true, created: [], kept: [...MIRROR_ENTITY], mapping: first.ok ? first.mapping : null }); + expect(fake.requests.filter((request) => request.method !== "GET")).toEqual([]); + }); + + it("recreates a deleted database alone, resets its pages, and repoints its dependents", async () => { + const mirrorId = await connect(); + const first = await ensureMirrorDatabases(mirrorId, { db, client: CLIENT }); + if (!first.ok) throw new Error("setup failed"); + const oldTools = first.mapping.tools!; + await upsertMirrorPage({ mirrorId, entity: "tools", entityId: crypto.randomUUID(), notionPageId: "old-tool-page", sourceUpdatedAt: null }, { db }); + await upsertMirrorPage({ mirrorId, entity: "units", entityId: crypto.randomUUID(), notionPageId: "unit-page", sourceUpdatedAt: null }, { db }); + await db.update(notionMirrors).set({ lastSyncedAt: new Date() }).where(eq(notionMirrors.id, mirrorId)); + + fake.databases.delete(oldTools); + fake.requests.length = 0; + const second = await ensureMirrorDatabases(mirrorId, { db, client: CLIENT }); + if (!second.ok) throw new Error(`expected ok, got ${second.code}`); + + expect(second.created).toEqual(["tools"]); + expect(second.kept).toEqual(["categories", "locations", "units", "resources", "maintenance", "projects"]); + const newTools = second.mapping.tools!; + expect(newTools).not.toBe(oldTools); + expect(await pagesFor(mirrorId, "tools")).toBe(0); + expect(await pagesFor(mirrorId, "units")).toBe(1); + expect((await getMirror(mirrorId, { db }))!.lastSyncedAt).toBeNull(); + + // Every database with a relation to tools now points at the new one. + const patched = fake.requests.filter((request) => request.method === "PATCH").map((request) => request.path); + expect(patched.sort()).toEqual( + [second.mapping.units, second.mapping.resources, second.mapping.maintenance, second.mapping.projects].map((id) => `/databases/${id}`).sort() + ); + for (const entity of ["units", "resources", "maintenance"] as const) { + expect(fake.databases.get(second.mapping[entity]!)!.properties.Tool.relation).toMatchObject({ database_id: newTools }); + } + expect(fake.databases.get(second.mapping.projects!)!.properties.Tools.relation).toMatchObject({ database_id: newTools }); + }); + + it("recreates a trashed database", async () => { + const mirrorId = await connect(); + const first = await ensureMirrorDatabases(mirrorId, { db, client: CLIENT }); + if (!first.ok) throw new Error("setup failed"); + const categories = fake.databases.get(first.mapping.categories!)!; + categories.archived = true; + categories.in_trash = true; + + const second = await ensureMirrorDatabases(mirrorId, { db, client: CLIENT }); + if (!second.ok) throw new Error(`expected ok, got ${second.code}`); + expect(second.created).toEqual(["categories"]); + expect(fake.databases.get(second.mapping.tools!)!.properties.Category.relation).toMatchObject({ + database_id: second.mapping.categories, + }); + }); + + it("saves a partial mapping when Notion fails half way, then finishes on the next press", async () => { + const mirrorId = await connect(); + // Categories and locations are created; the third create (tools) hits a 503. + let creates = 0; + const original = fake.handle; + fake.handle = (method, path, headers, body) => { + if (method === "POST" && path.endsWith("/databases") && ++creates === 3) { + return { status: 503, headers: { "Content-Type": "application/json" }, body: { object: "error", status: 503, code: "service_unavailable", message: "Down." } }; + } + return original(method, path, headers, body); + }; + + const partial = await ensureMirrorDatabases(mirrorId, { db, client: CLIENT }); + expect(partial).toEqual({ ok: false, code: "notion_unavailable", created: ["categories", "locations"], entity: "tools" }); + expect(Object.keys((await getMirror(mirrorId, { db }))!.mapping)).toEqual(["categories", "locations"]); + + const rest = await ensureMirrorDatabases(mirrorId, { db, client: CLIENT }); + if (!rest.ok) throw new Error(`expected ok, got ${rest.code}`); + expect(rest.kept).toEqual(["categories", "locations"]); + expect(rest.created).toEqual(["tools", "units", "resources", "maintenance", "projects"]); + }); + + it("says unauthorized on a 401, page_not_found for an unshared page, and needs a connection", async () => { + const revoked = await connect("ntn_revokedTOKEN"); + expect(await ensureMirrorDatabases(revoked, { db, client: CLIENT })).toEqual({ + ok: false, + code: "unauthorized", + created: [], + entity: "categories", + }); + + const unshared = await connect(TOKEN, "0f5e4a3c-9999-2222-3333-44445555aaaa"); + expect(await ensureMirrorDatabases(unshared, { db, client: CLIENT })).toMatchObject({ ok: false, code: "page_not_found" }); + + expect(await ensureMirrorDatabases(crypto.randomUUID(), { db, client: CLIENT })).toMatchObject({ ok: false, code: "not_connected" }); + + vi.stubEnv("AUTH_SECRET", ""); + expect(await ensureMirrorDatabases(unshared, { db, client: CLIENT })).toMatchObject({ ok: false, code: "key_unavailable" }); + }); +}); + +describe("applyPastedMapping", () => { + async function databasesFromAnotherMirror() { + const source = await connect(); + const result = await ensureMirrorDatabases(source, { db, client: CLIENT }); + if (!result.ok) throw new Error("setup failed"); + return result.mapping; + } + + it("persists pasted ids that validate — URLs and bare ids alike — and resets changed entities", async () => { + const existing = await databasesFromAnotherMirror(); + const mirrorId = await connect(); + await upsertMirrorPage({ mirrorId, entity: "tools", entityId: crypto.randomUUID(), notionPageId: "stale", sourceUpdatedAt: null }, { db }); + + const pasted = { + categories: existing.categories!.replace(/-/g, ""), + locations: `https://www.notion.so/workspace/Locations-${existing.locations!.replace(/-/g, "")}?v=abc`, + tools: existing.tools!, + }; + const result = await applyPastedMapping(mirrorId, pasted, { db, client: CLIENT }); + expect(result).toEqual({ + ok: true, + mapping: { categories: existing.categories, locations: existing.locations, tools: existing.tools }, + }); + expect((await getMirror(mirrorId, { db }))!.mapping).toEqual(result.ok ? result.mapping : null); + expect(await pagesFor(mirrorId, "tools")).toBe(0); + }); + + it("persists nothing when any pasted database does not match", async () => { + const existing = await databasesFromAnotherMirror(); + const mirrorId = await connect(); + delete fake.databases.get(existing.tools!)!.properties["Emergency stop"]; + fake.databases.get(existing.tools!)!.properties.Published.type = "rich_text"; + + const result = await applyPastedMapping( + mirrorId, + { categories: existing.categories!, tools: existing.tools!, units: "not an id", resources: crypto.randomUUID() }, + { db, client: CLIENT } + ); + // An unreadable id is refused before Notion is asked anything. + expect(result).toEqual({ ok: false, code: "invalid_database_id", problems: [{ entity: "units", code: "invalid_database_id" }] }); + + const second = await applyPastedMapping( + mirrorId, + { categories: existing.categories!, tools: existing.tools!, resources: crypto.randomUUID() }, + { db, client: CLIENT } + ); + expect(second).toEqual({ + ok: false, + code: "schema_mismatch", + problems: [ + { entity: "tools", code: "schema_mismatch", missing: ["Emergency stop"], wrongType: ["Published"] }, + { entity: "resources", code: "database_not_found" }, + ], + }); + expect((await getMirror(mirrorId, { db }))!.mapping).toEqual({}); + }); + + it("checks a pasted relation against the database its target maps to", async () => { + const existing = await databasesFromAnotherMirror(); + const other = await databasesFromAnotherMirror(); + const mirrorId = await connect(); + const result = await applyPastedMapping(mirrorId, { categories: other.categories!, tools: existing.tools! }, { db, client: CLIENT }); + expect(result).toMatchObject({ ok: false, code: "schema_mismatch", problems: [{ entity: "tools", wrongType: ["Category"] }] }); + }); + + it("says unauthorized on a 401", async () => { + const existing = await databasesFromAnotherMirror(); + const mirrorId = await connect("ntn_revokedTOKEN"); + expect(await applyPastedMapping(mirrorId, { tools: existing.tools! }, { db, client: CLIENT })).toEqual({ + ok: false, + code: "unauthorized", + problems: [], + }); + }); +}); diff --git a/v5/src/lib/mirror/databases.ts b/v5/src/lib/mirror/databases.ts new file mode 100644 index 0000000..ad56f99 --- /dev/null +++ b/v5/src/lib/mirror/databases.ts @@ -0,0 +1,238 @@ +import { getDb } from "../db/client.ts"; +import { MIRROR_ENTITY, type MirrorEntity } from "../db/schema/vocabulary.ts"; +import type { Db } from "../db/types.ts"; +import { getMirror, getMirrorTokenCiphertext, resetMirrorEntities, setMirrorMapping } from "../data/mirrors.ts"; +import { mirrorClientFor } from "./credentials.ts"; +import { + databaseCreateBody, + expectedProperties, + normalizeDatabaseId, + propertySchemaBody, + validateDatabaseSchema, +} from "./database-schemas.ts"; +import { NotionMirrorError, type NotionClient, type NotionClientOptions, type NotionDatabaseObject } from "./notion-client.ts"; +import { parseNotionId } from "./notion-id.ts"; +import type { MappingProblem, MirrorMapping, MirrorSetupError } from "./types.ts"; + +/** + * Setting up a mirror's databases (spec §3.8 "Mapping", §5.8). + * + * - {@link ensureMirrorDatabases} is **Create databases**: it keeps every mapped + * database that still exists and creates the rest under the connected page, + * in dependency order, so each relation can point at a database that is + * already there. Run it again after somebody deletes a database by hand and + * it recreates only that one (§5.8). + * - {@link applyPastedMapping} is the other road: an admin who already has + * databases pastes their ids, and each is checked against the fixed schema + * before anything is saved. + * + * Both return refusals as values (`MirrorSetupError` codes the page + * translates), never a Notion message: those can carry text from the + * workspace, and nothing from Notion reaches the page except its title. + * + * Relative imports with `.ts` extensions, no `server-only`: reachable from + * step code. + */ + +export interface MirrorDatabaseOptions { + db?: Db; + client?: Partial>; +} + +export type EnsureDatabasesResult = + | { ok: true; created: MirrorEntity[]; kept: MirrorEntity[]; mapping: MirrorMapping } + | { ok: false; code: MirrorSetupError; created: MirrorEntity[]; entity: MirrorEntity | null }; + +export type ApplyMappingResult = + | { ok: true; mapping: MirrorMapping } + | { ok: false; code: MirrorSetupError; problems: MappingProblem[] }; + +type ClientResult = { ok: true; client: NotionClient; parentPageId: string; mapping: MirrorMapping } | { ok: false; code: MirrorSetupError }; + +async function openMirror(mirrorId: string, options: MirrorDatabaseOptions, db: Db): Promise { + const mirror = await getMirror(mirrorId, { db }); + if (!mirror) return { ok: false, code: "not_connected" }; + const ciphertext = await getMirrorTokenCiphertext(mirrorId, { db }); + const opened = mirrorClientFor(ciphertext, options.client ?? {}); + if (!opened.ok) return { ok: false, code: opened.code }; + return { ok: true, client: opened.client, parentPageId: mirror.parentPageId, mapping: mirror.mapping }; +} + +/** A Notion failure during setup, as the code the page shows. */ +export function setupErrorFor(error: unknown, notFound: MirrorSetupError): MirrorSetupError { + if (!(error instanceof NotionMirrorError)) return "notion_unavailable"; + switch (error.code) { + case "unauthorized": + case "restricted": + return "unauthorized"; + case "page_not_found": + case "database_not_found": + case "not_found": + return notFound; + default: + return "notion_unavailable"; + } +} + +function isGone(database: NotionDatabaseObject): boolean { + return database.archived === true || database.in_trash === true; +} + +/** + * `PATCH` body for a kept database whose relations do not point at the + * databases `resolved` now holds — the dependents of a recreated database. + * Null when every relation is already right. + */ +function relationRepair(entity: MirrorEntity, database: NotionDatabaseObject, resolved: MirrorMapping): Record | null { + const properties: Record = {}; + for (const expected of expectedProperties(entity, resolved)) { + if (expected.type !== "relation" || !expected.targetDatabaseId) continue; + const actual = database.properties?.[expected.name]; + if (actual && actual.type !== "relation") continue; // a clash of types is for the admin to see, not for us to overwrite + if (actual && normalizeDatabaseId(actual.relation?.database_id) === expected.targetDatabaseId) continue; + const body = propertySchemaBody(expected, expected.targetDatabaseId); + if (body) properties[expected.name] = body; + } + return Object.keys(properties).length ? { properties } : null; +} + +/** + * Keep what exists, create what does not, fix the relations of what was kept, + * and save the mapping — including a partial one when Notion fails half way, + * so a second press picks up from there instead of creating duplicates. + * + * A database is kept when a `GET` answers 200 and it is neither archived nor + * in the trash; a 404, archived or trashed database is created again. Every + * entity created loses its `mirror_pages` rows and makes the next push a full + * one (`resetMirrorEntities`): its old page ids point into a database that is + * gone, and a newly mapped entity has rows older than `last_synced_at`. The + * reset also marks the pages that link to it (units, resources, maintenance + * and projects for tools, and so on) as not mirrored, so the push rewrites + * their relations to the new pages. + */ +export async function ensureMirrorDatabases( + mirrorId: string, + options: MirrorDatabaseOptions = {} +): Promise { + const db = options.db ?? (await getDb()); + const opened = await openMirror(mirrorId, options, db); + if (!opened.ok) return { ok: false, code: opened.code, created: [], entity: null }; + const { client, parentPageId } = opened; + + const resolved: MirrorMapping = {}; + const created: MirrorEntity[] = []; + const kept: MirrorEntity[] = []; + let failure: { code: MirrorSetupError; entity: MirrorEntity } | null = null; + + for (const entity of MIRROR_ENTITY) { + const mappedId = opened.mapping[entity]; + try { + let existing: NotionDatabaseObject | null = null; + if (mappedId) { + try { + existing = await client.getDatabase(mappedId); + if (isGone(existing)) existing = null; + } catch (error) { + if (!(error instanceof NotionMirrorError) || error.code !== "database_not_found") throw error; + } + } + + if (existing) { + const id = normalizeDatabaseId(existing.id) ?? mappedId!; + const repair = relationRepair(entity, existing, resolved); + if (repair) await client.updateDatabase(id, repair); + resolved[entity] = id; + kept.push(entity); + continue; + } + + let fresh: NotionDatabaseObject; + try { + fresh = await client.createDatabase(databaseCreateBody(entity, parentPageId, resolved)); + } catch (error) { + // A 404 on create is the parent page: unshared, or deleted. + failure = { code: setupErrorFor(error, "page_not_found"), entity }; + break; + } + resolved[entity] = normalizeDatabaseId(fresh.id) ?? fresh.id; + created.push(entity); + } catch (error) { + failure = { code: setupErrorFor(error, "database_not_found"), entity }; + break; + } + } + + // Save what is true now, even half way: databases that were created exist + // in Notion, and forgetting them would duplicate them on the next press. + const mapping: MirrorMapping = failure ? { ...opened.mapping, ...resolved } : resolved; + if (created.length) await resetMirrorEntities(mirrorId, created, { db }); + const saved = await setMirrorMapping(mirrorId, mapping, { db }); + + if (failure) return { ok: false, code: failure.code, created, entity: failure.entity }; + return { ok: true, created, kept, mapping: saved.mapping }; +} + +/** + * Validate pasted database ids and, only when every one passes, save them + * over the existing mapping. + * + * Each value may be a bare id, a dashed uuid or a Notion URL. A value that is + * not one is `invalid_database_id`; a database Notion cannot find (or that is + * archived) is `database_not_found`; one without the expected properties is + * `schema_mismatch`, with which properties are missing or of the wrong type. A + * relation is checked against the database its target will map to once this + * paste is saved. Blank values are ignored — they leave that entity as it is. + * + * An entity whose database changes loses its `mirror_pages` rows, and the next + * push is a full one. + */ +export async function applyPastedMapping( + mirrorId: string, + pasted: Partial>, + options: MirrorDatabaseOptions = {} +): Promise { + const db = options.db ?? (await getDb()); + const problems: MappingProblem[] = []; + const parsed: MirrorMapping = {}; + for (const entity of MIRROR_ENTITY) { + const value = pasted[entity]; + if (typeof value !== "string" || value.trim() === "") continue; + const id = parseNotionId(value); + if (id) parsed[entity] = id; + else problems.push({ entity, code: "invalid_database_id" }); + } + if (problems.length) return { ok: false, code: "invalid_database_id", problems }; + if (Object.keys(parsed).length === 0) return { ok: false, code: "invalid_database_id", problems: [] }; + + const opened = await openMirror(mirrorId, options, db); + if (!opened.ok) return { ok: false, code: opened.code, problems: [] }; + const merged: MirrorMapping = { ...opened.mapping, ...parsed }; + + for (const entity of MIRROR_ENTITY) { + const id = parsed[entity]; + if (!id) continue; + let database: NotionDatabaseObject; + try { + database = await opened.client.getDatabase(id); + } catch (error) { + const code = setupErrorFor(error, "database_not_found"); + if (code !== "database_not_found") return { ok: false, code, problems: [] }; + problems.push({ entity, code: "database_not_found" }); + continue; + } + if (isGone(database)) { + problems.push({ entity, code: "database_not_found" }); + continue; + } + const problem = validateDatabaseSchema(entity, database, merged); + if (problem) problems.push(problem); + } + if (problems.length) return { ok: false, code: problems[0].code, problems }; + + const changed = MIRROR_ENTITY.filter( + (entity) => parsed[entity] && normalizeDatabaseId(opened.mapping[entity]) !== parsed[entity] + ); + if (changed.length) await resetMirrorEntities(mirrorId, changed, { db }); + const saved = await setMirrorMapping(mirrorId, merged, { db }); + return { ok: true, mapping: saved.mapping }; +} diff --git a/v5/src/lib/mirror/limits.ts b/v5/src/lib/mirror/limits.ts new file mode 100644 index 0000000..58a545b --- /dev/null +++ b/v5/src/lib/mirror/limits.ts @@ -0,0 +1,57 @@ +/** + * The Notion mirror's numbers (spec §3.8, §8), in one place so the push, the + * claims in `data/mirrors.ts` and the page's copy cannot disagree. + * + * Relative imports only (none here): workflow step code loads this under plain + * Node. + */ + +/** One push stops starting new Notion calls after this long; the rest waits for the next push. */ +export const MIRROR_PUSH_BUDGET_MS = 45_000; + +/** Notion's documented average rate limit is 3 requests per second per integration. */ +export const MIRROR_REQUESTS_PER_SECOND = 3; + +/** How many 429s one request may wait out before it gives up as `rate_limited`. */ +export const MIRROR_MAX_RATE_LIMIT_RETRIES = 3; + +/** A `running_since` older than this is a push that died; the overlap guard lets the next one in. */ +export const MIRROR_RUN_STALE_MINUTES = 15; + +/** Sync now: one push per mirror per this many minutes (§8 rate limiting). */ +export const MIRROR_SYNC_NOW_MINUTES = 15; + +/** How long `mirrorPushAfterChange` sleeps so a burst of edits becomes one push (§3.8 trigger 1). */ +export const MIRROR_COALESCE_DELAY = "2m"; + +/** A `push_requested_at` older than this is a coalescing run that never finished; a new change may claim again. */ +export const MIRROR_COALESCE_STALE_MINUTES = 10; + +/** + * The watermark a push advances `last_synced_at` to is its claim time minus + * this. A row committed by a transaction that began before the claim but + * committed after the push read its table carries an `updated_at` before the + * claim; the margin makes the next push select it again. Re-pushing a few + * unchanged rows is harmless; missing one is not. + */ +export const MIRROR_WATERMARK_SAFETY_MINUTES = 5; + +/** A workflow pushes an `incomplete` mirror again at most this many times before leaving it to the next trigger. */ +export const MIRROR_MAX_ROUNDS = 6; + +/** + * A round skipped because another push holds the mirror waits + * {@link MIRROR_BUSY_PAUSE} and tries again, at most this many times, so a + * change or a backstop that arrives mid-push is still pushed once that push is + * done — whose table reads may have been taken before the change landed. + */ +export const MIRROR_BUSY_RETRIES = 10; + +/** How long a round skipped as `running` waits before trying the claim again. */ +export const MIRROR_BUSY_PAUSE = "30s"; + +/** `maxRetries` on each mirror step function. */ +export const MIRROR_STEP_MAX_RETRIES = 2; + +/** How often `/admin/mirror` polls while a push is running or pending. */ +export const MIRROR_POLL_INTERVAL_MS = 5_000; diff --git a/v5/src/lib/mirror/notion-client.test.ts b/v5/src/lib/mirror/notion-client.test.ts new file mode 100644 index 0000000..21f825a --- /dev/null +++ b/v5/src/lib/mirror/notion-client.test.ts @@ -0,0 +1,395 @@ +// @vitest-environment node +import { http, HttpResponse } from "msw"; +import { server } from "../../../test/msw/server"; +import { + NOTION_API_VERSION, + NOTION_DEFAULT_BASE_URL, + NotionMirrorError, + createNotionClient, + notionApiBaseUrl, + pageTitle, + scrubSecrets, + type NotionPageObject, +} from "./notion-client"; + +/** + * The mirror's Notion client against MSW on `api.notion.com` (spec §3.8, §8, + * §10 "429 with Retry-After"). Time is injected: `now` reads a counter and + * `sleep` advances it, so throttling and backoff are asserted exactly and no + * test waits on a real clock. + */ +const TOKEN = "ntn_TESTtoken0123456789abcdefABCDEF"; +const BASE = NOTION_DEFAULT_BASE_URL; +const PAGE_ID = "0f5e4a3c-1111-2222-3333-44445555aaaa"; + +function fakeClock(start = 1_000_000) { + let t = start; + const sleeps: number[] = []; + return { + now: () => t, + sleep: async (ms: number) => { + sleeps.push(ms); + t += ms; + }, + sleeps, + advance: (ms: number) => { + t += ms; + }, + }; +} + +function page(id = PAGE_ID): NotionPageObject { + return { + object: "page", + id, + properties: { title: { id: "title", type: "title", title: [{ plain_text: "MakerLab Tools " }, { plain_text: "— mirror" }] } }, + }; +} + +function notionError(status: number, code: string, message: string, headers: Record = {}) { + return HttpResponse.json({ object: "error", status, code, message }, { status, headers }); +} + +async function caught(promise: Promise): Promise { + try { + await promise; + } catch (error) { + expect(error).toBeInstanceOf(NotionMirrorError); + return error as NotionMirrorError; + } + throw new Error("expected a NotionMirrorError"); +} + +describe("createNotionClient", () => { + it("sends the bearer token, the pinned Notion-Version and JSON", async () => { + let seen: Headers | null = null; + let body: unknown = null; + server.use( + http.patch(`${BASE}/pages/:id`, async ({ request, params }) => { + seen = request.headers; + body = await request.json(); + return HttpResponse.json({ object: "page", id: params.id }); + }) + ); + const clock = fakeClock(); + const client = createNotionClient({ token: TOKEN, now: clock.now, sleep: clock.sleep }); + + await expect(client.updatePage(PAGE_ID, { archived: true })).resolves.toEqual({ object: "page", id: PAGE_ID }); + expect(seen!.get("authorization")).toBe(`Bearer ${TOKEN}`); + expect(seen!.get("notion-version")).toBe(NOTION_API_VERSION); + expect(NOTION_API_VERSION).toBe("2022-06-28"); + expect(seen!.get("content-type")).toBe("application/json"); + expect(body).toEqual({ archived: true }); + }); + + it("spaces request starts at least 1000 / requestsPerSecond ms apart", async () => { + const clock = fakeClock(); + const starts: number[] = []; + server.use( + http.get(`${BASE}/pages/:id`, ({ params }) => { + starts.push(clock.now()); + return HttpResponse.json(page(params.id as string)); + }) + ); + const client = createNotionClient({ token: TOKEN, now: clock.now, sleep: clock.sleep, requestsPerSecond: 3 }); + + for (let i = 0; i < 4; i += 1) await client.getPage(PAGE_ID); + + const gaps = starts.slice(1).map((start, i) => start - starts[i]); + for (const gap of gaps) expect(gap).toBeGreaterThanOrEqual(1000 / 3 - 1e-6); + }); + + it("queues concurrent callers instead of bursting", async () => { + // A clock that stands still while the callers queue, so each one's wait is + // measured from the same instant. + const sleeps: number[] = []; + server.use(http.get(`${BASE}/pages/:id`, ({ params }) => HttpResponse.json(page(params.id as string)))); + const client = createNotionClient({ + token: TOKEN, + now: () => 5_000, + sleep: async (ms) => { + sleeps.push(ms); + }, + requestsPerSecond: 2, + }); + + await Promise.all([client.getPage(PAGE_ID), client.getPage(PAGE_ID), client.getPage(PAGE_ID)]); + + // Slots reserved at +0, +500 and +1000. + expect(sleeps).toEqual([500, 1000]); + }); + + it("waits out a 429 for Retry-After seconds and then succeeds", async () => { + const clock = fakeClock(); + let calls = 0; + server.use( + http.get(`${BASE}/databases/:id`, ({ params }) => { + calls += 1; + if (calls === 1) return notionError(429, "rate_limited", "Slow down.", { "Retry-After": "2" }); + return HttpResponse.json({ object: "database", id: params.id, properties: {} }); + }) + ); + const client = createNotionClient({ token: TOKEN, now: clock.now, sleep: clock.sleep }); + + await expect(client.getDatabase(PAGE_ID)).resolves.toMatchObject({ object: "database", id: PAGE_ID }); + expect(calls).toBe(2); + expect(clock.sleeps).toEqual([2000]); + }); + + it("defaults a 429 without Retry-After to one second", async () => { + const clock = fakeClock(); + let calls = 0; + server.use( + http.post(`${BASE}/pages`, () => { + calls += 1; + return calls === 1 ? notionError(429, "rate_limited", "Slow down.") : HttpResponse.json({ object: "page", id: PAGE_ID }); + }) + ); + const client = createNotionClient({ token: TOKEN, now: clock.now, sleep: clock.sleep }); + + await client.createPage({ parent: { database_id: PAGE_ID }, properties: {} }); + expect(clock.sleeps).toEqual([1000]); + }); + + it("gives up as rate_limited once the retries are spent", async () => { + const clock = fakeClock(); + let calls = 0; + server.use( + http.get(`${BASE}/pages/:id`, () => { + calls += 1; + return notionError(429, "rate_limited", "Slow down.", { "Retry-After": "1" }); + }) + ); + const client = createNotionClient({ token: TOKEN, now: clock.now, sleep: clock.sleep, maxRateLimitRetries: 2 }); + + const error = await caught(client.getPage(PAGE_ID)); + expect(error.code).toBe("rate_limited"); + expect(error.status).toBe(429); + expect(calls).toBe(3); + }); + + it("throws rate_limited without waiting when Retry-After runs past the deadline", async () => { + const clock = fakeClock(); + server.use( + http.get(`${BASE}/pages/:id`, () => notionError(429, "rate_limited", "Slow down.", { "Retry-After": "30" })) + ); + const client = createNotionClient({ + token: TOKEN, + now: clock.now, + sleep: clock.sleep, + deadline: clock.now() + 10_000, + }); + + const error = await caught(client.getPage(PAGE_ID)); + expect(error.code).toBe("rate_limited"); + expect(clock.sleeps).toEqual([]); + }); + + it("starts nothing after the deadline", async () => { + const clock = fakeClock(); + let calls = 0; + server.use( + http.get(`${BASE}/pages/:id`, () => { + calls += 1; + return HttpResponse.json(page()); + }) + ); + const client = createNotionClient({ token: TOKEN, now: clock.now, sleep: clock.sleep, deadline: clock.now() + 500 }); + + expect(client.remainingMs()).toBe(500); + await client.getPage(PAGE_ID); + clock.advance(600); + expect(client.remainingMs()).toBeLessThanOrEqual(0); + + const error = await caught(client.getPage(PAGE_ID)); + expect(error.code).toBe("deadline"); + expect(calls).toBe(1); + }); + + it("lets a request started just before the deadline finish, rather than abort a create Notion already has", async () => { + // Real time for the fetch, fake time for the budget: 5 ms left when the + // POST starts, and Notion takes 150 ms to answer. Cutting the request to + // the leftover budget would abort it after Notion made the page, and the + // next round would create it again. + const clock = fakeClock(); + let created = 0; + server.use( + http.post(`${BASE}/pages`, async () => { + created += 1; + await new Promise((resolve) => setTimeout(resolve, 150)); + return HttpResponse.json(page()); + }) + ); + const client = createNotionClient({ token: TOKEN, now: clock.now, sleep: clock.sleep, deadline: clock.now() + 5 }); + + await expect(client.createPage({ parent: { database_id: "d" }, properties: {} })).resolves.toMatchObject({ id: PAGE_ID }); + expect(created).toBe(1); + }); + + it("reports Infinity remaining with no deadline", () => { + expect(createNotionClient({ token: TOKEN }).remainingMs()).toBe(Number.POSITIVE_INFINITY); + }); + + describe("maps Notion's statuses", () => { + it("401 → unauthorized", async () => { + server.use(http.get(`${BASE}/pages/:id`, () => notionError(401, "unauthorized", "API token is invalid."))); + const error = await caught(createNotionClient({ token: TOKEN }).getPage(PAGE_ID)); + expect(error.code).toBe("unauthorized"); + expect(error.status).toBe(401); + expect(error.notionCode).toBe("unauthorized"); + expect(error.message).toBe("Notion 401 unauthorized: API token is invalid."); + }); + + it("403 → restricted", async () => { + server.use(http.get(`${BASE}/pages/:id`, () => notionError(403, "restricted_resource", "No access."))); + expect((await caught(createNotionClient({ token: TOKEN }).getPage(PAGE_ID))).code).toBe("restricted"); + }); + + it("getDatabase 404 → database_not_found", async () => { + server.use( + http.get(`${BASE}/databases/:id`, () => + notionError(404, "object_not_found", `Could not find database with ID: ${PAGE_ID}.`) + ) + ); + const error = await caught(createNotionClient({ token: TOKEN }).getDatabase(PAGE_ID)); + expect(error.code).toBe("database_not_found"); + expect(error.message).toMatch(/^Notion 404 object_not_found: Could not find database/); + }); + + it("getPage 404 → page_not_found, createDatabase 404 → page_not_found (the parent)", async () => { + server.use( + http.get(`${BASE}/pages/:id`, () => notionError(404, "object_not_found", "Not found.")), + http.post(`${BASE}/databases`, () => notionError(404, "object_not_found", "Not found.")) + ); + const client = createNotionClient({ token: TOKEN }); + expect((await caught(client.getPage(PAGE_ID))).code).toBe("page_not_found"); + expect((await caught(client.createDatabase({}))).code).toBe("page_not_found"); + }); + + it("updatePage 404 → page_not_found, createPage 404 → database_not_found", async () => { + server.use( + http.patch(`${BASE}/pages/:id`, () => notionError(404, "object_not_found", "Not found.")), + http.post(`${BASE}/pages`, () => notionError(404, "object_not_found", "Not found.")) + ); + const client = createNotionClient({ token: TOKEN }); + expect((await caught(client.updatePage(PAGE_ID, {}))).code).toBe("page_not_found"); + expect((await caught(client.createPage({}))).code).toBe("database_not_found"); + }); + + it("updateDatabase 404 → database_not_found", async () => { + server.use(http.patch(`${BASE}/databases/:id`, () => notionError(404, "object_not_found", "Not found."))); + expect((await caught(createNotionClient({ token: TOKEN }).updateDatabase(PAGE_ID, {}))).code).toBe( + "database_not_found" + ); + }); + + it("400 → validation, 409 → conflict", async () => { + server.use( + http.post(`${BASE}/pages`, () => notionError(400, "validation_error", "body.properties.Name should be defined.")), + http.patch(`${BASE}/pages/:id`, () => notionError(409, "conflict_error", "Conflict occurred while saving.")) + ); + const client = createNotionClient({ token: TOKEN }); + expect((await caught(client.createPage({}))).code).toBe("validation"); + expect((await caught(client.updatePage(PAGE_ID, {}))).code).toBe("conflict"); + }); + + it("500 → unavailable, and a non-JSON body is kept as text", async () => { + server.use(http.get(`${BASE}/pages/:id`, () => new HttpResponse("upstream sad", { status: 502 }))); + const error = await caught(createNotionClient({ token: TOKEN }).getPage(PAGE_ID)); + expect(error.code).toBe("unavailable"); + expect(error.message).toBe("Notion 502: upstream sad"); + }); + + it("a network failure → unavailable", async () => { + server.use(http.get(`${BASE}/pages/:id`, () => HttpResponse.error())); + expect((await caught(createNotionClient({ token: TOKEN }).getPage(PAGE_ID))).code).toBe("unavailable"); + }); + + it("cuts Notion's message to 300 characters", async () => { + server.use(http.post(`${BASE}/pages`, () => notionError(400, "validation_error", "x".repeat(1000)))); + const error = await caught(createNotionClient({ token: TOKEN }).createPage({})); + expect(error.message).toBe(`Notion 400 validation_error: ${"x".repeat(300)}`); + }); + }); + + it("never puts the token in an error message or on the console", async () => { + const spies = (["log", "info", "warn", "error", "debug"] as const).map((method) => + vi.spyOn(console, method).mockImplementation(() => {}) + ); + server.use( + http.get(`${BASE}/pages/:id`, () => + notionError(401, "unauthorized", `API token ${TOKEN} is invalid; also secret_ABC123def and reporter ada@cornell.edu`) + ), + http.post(`${BASE}/pages`, () => new HttpResponse(`echo: Bearer ${TOKEN}`, { status: 500 })), + http.get(`${BASE}/databases/:id`, () => HttpResponse.error()) + ); + const client = createNotionClient({ token: TOKEN }); + + const errors = [ + await caught(client.getPage(PAGE_ID)), + await caught(client.createPage({ secret: TOKEN })), + await caught(client.getDatabase(PAGE_ID)), + ]; + for (const error of errors) { + const text = `${error.message} ${error.stack ?? ""} ${JSON.stringify(error)}`; + expect(text).not.toContain(TOKEN); + expect(text).not.toContain("secret_ABC123def"); + expect(text).not.toContain("ada@cornell.edu"); + } + expect(errors[0].message).toContain("[redacted]"); + + for (const spy of spies) { + for (const call of spy.mock.calls) expect(call.map(String).join(" ")).not.toContain(TOKEN); + } + }); +}); + +describe("notionApiBaseUrl", () => { + it("defaults to api.notion.com", () => { + expect(notionApiBaseUrl()).toBe("https://api.notion.com/v1"); + }); + + it("reads NOTION_API_BASE_URL at call time", async () => { + vi.stubEnv("NOTION_API_BASE_URL", "http://127.0.0.1:3102/v1/"); + expect(notionApiBaseUrl()).toBe("http://127.0.0.1:3102/v1"); + + server.use( + http.get("http://127.0.0.1:3102/v1/pages/:id", ({ params }) => HttpResponse.json(page(params.id as string))) + ); + const client = createNotionClient({ token: TOKEN }); + await expect(client.getPage(PAGE_ID)).resolves.toMatchObject({ id: PAGE_ID }); + }); +}); + +describe("pageTitle", () => { + it("joins the plain text of the title property, whatever it is called", () => { + expect(pageTitle(page())).toBe("MakerLab Tools — mirror"); + expect( + pageTitle({ + object: "page", + id: PAGE_ID, + properties: { + Status: { type: "select" }, + Name: { type: "title", title: [{ plain_text: "Form 4" }] }, + }, + }) + ).toBe("Form 4"); + }); + + it("is null for an untitled page", () => { + expect(pageTitle({ object: "page", id: PAGE_ID, properties: { title: { type: "title", title: [] } } })).toBeNull(); + expect(pageTitle({ object: "page", id: PAGE_ID, properties: {} })).toBeNull(); + }); +}); + +describe("scrubSecrets", () => { + it("removes the given secrets and anything token-shaped", () => { + expect(scrubSecrets("a hunter2 b", ["hunter2"])).toBe("a [redacted] b"); + expect(scrubSecrets("key secret_abcDEF123 and ntn_xyz789")).toBe("key [redacted] and [redacted]"); + expect(scrubSecrets("nothing here", [""])).toBe("nothing here"); + }); + + it("removes email addresses", () => { + expect(scrubSecrets("reported by ada.l@cornell.edu today")).toBe("reported by [email] today"); + }); +}); diff --git a/v5/src/lib/mirror/notion-client.ts b/v5/src/lib/mirror/notion-client.ts new file mode 100644 index 0000000..a736de7 --- /dev/null +++ b/v5/src/lib/mirror/notion-client.ts @@ -0,0 +1,327 @@ +import { + MIRROR_MAX_RATE_LIMIT_RETRIES, + MIRROR_REQUESTS_PER_SECOND, +} from "./limits.ts"; + +/** + * The mirror's Notion client (spec §3.8 "Push", §8 "External calls"): raw + * `fetch` against the 2022-06-28 API, no SDK. + * + * What it guarantees, per client instance: + * + * - **Throttled.** Request *starts* are spaced at least `1000 / requestsPerSecond` + * ms apart (3/s by default — Notion's documented average). The slot is + * reserved synchronously, so concurrent callers queue rather than burst. + * - **429 is waited out** for `Retry-After` seconds (1 when absent), up to + * `maxRateLimitRetries` times. A wait that would pass the deadline is not + * started: the call throws `rate_limited` at once, and what is left waits for + * the next push. + * - **Bounded.** With a `deadline`, no request *starts* after it (`deadline`). + * A request already started is never cut short by the deadline: aborting a + * `POST /pages` Notion has already received would create the page and lose + * its id, and the next push would create it again. Every fetch instead gets + * the fixed {@link NOTION_REQUEST_TIMEOUT_MS} ceiling (`unavailable` when it + * fires), so a push step ends at most that long after its 45 s budget — far + * inside the 300 s function limit — and a setup call cannot hang a server + * action. + * - **Never leaks the token.** Errors are built from the status, Notion's error + * code and Notion's message run through {@link scrubSecrets} with the token, + * cut to 300 characters. Headers and request bodies are never included, and + * nothing here logs. + * + * Relative imports with `.ts` extensions, no `server-only`: step code runs this + * from an esbuild bundle under plain Node. MSW intercepts its `fetch` in tests. + */ + +export const NOTION_API_VERSION = "2022-06-28"; +export const NOTION_DEFAULT_BASE_URL = "https://api.notion.com/v1"; + +/** Every request is aborted after this long, deadline or not (a request never outlives it by the budget). */ +export const NOTION_REQUEST_TIMEOUT_MS = 30_000; + +/** How much of Notion's own message an error keeps. */ +const MAX_MESSAGE_LENGTH = 300; + +/** + * The API base URL, read at call time. `NOTION_API_BASE_URL` is a test-only + * override (the E2E stub server); production never sets it. + */ +export function notionApiBaseUrl(): string { + const override = process.env.NOTION_API_BASE_URL?.trim(); + return (override || NOTION_DEFAULT_BASE_URL).replace(/\/+$/, ""); +} + +export type NotionErrorCode = + | "unauthorized" + | "restricted" + | "page_not_found" + | "database_not_found" + | "not_found" + | "validation" + | "conflict" + | "rate_limited" + | "unavailable" + | "deadline"; + +/** + * Every failure the client reports. `message` reads like + * `Notion 404 object_not_found: Could not find database with ID: …` and never + * holds the token. + */ +export class NotionMirrorError extends Error { + code: NotionErrorCode; + status: number | null; + notionCode: string | null; + + constructor(code: NotionErrorCode, message: string, status: number | null = null, notionCode: string | null = null) { + super(message); + this.name = "NotionMirrorError"; + this.code = code; + this.status = status; + this.notionCode = notionCode; + } +} + +export interface NotionClientOptions { + token: string; + /** Epoch ms, compared against `now()`. No request starts after it. */ + deadline?: number; + requestsPerSecond?: number; + maxRateLimitRetries?: number; + now?: () => number; + sleep?: (ms: number) => Promise; + /** Overrides {@link notionApiBaseUrl}. */ + baseUrl?: string; +} + +export interface NotionPageObject { + object: "page"; + id: string; + archived?: boolean; + in_trash?: boolean; + properties: Record< + string, + { id?: string; type: string; title?: { plain_text?: string }[] } & Record + >; +} + +export interface NotionDatabaseObject { + object: "database"; + id: string; + archived?: boolean; + in_trash?: boolean; + title?: { plain_text?: string }[]; + properties: Record< + string, + { id?: string; name?: string; type: string; relation?: { database_id?: string } } & Record + >; +} + +export interface NotionClient { + /** `GET /pages/:id`; a 404 is `page_not_found`. */ + getPage(id: string): Promise; + /** `GET /databases/:id`; a 404 is `database_not_found`. */ + getDatabase(id: string): Promise; + /** `POST /databases`; a 404 is `page_not_found` — the parent page. */ + createDatabase(body: unknown): Promise; + /** `PATCH /databases/:id`; a 404 is `database_not_found`. */ + updateDatabase(id: string, body: unknown): Promise; + /** `POST /pages`; a 404 is `database_not_found` — the parent database. */ + createPage(body: unknown): Promise<{ id: string }>; + /** `PATCH /pages/:id`; a 404 is `page_not_found`. */ + updatePage(id: string, body: unknown): Promise<{ id: string }>; + /** Milliseconds left before the deadline; `Infinity` with none. */ + remainingMs(): number; +} + +type NotFoundCode = "page_not_found" | "database_not_found" | "not_found"; + +export function createNotionClient(options: NotionClientOptions): NotionClient { + const { token, deadline } = options; + const now = options.now ?? Date.now; + const sleep = options.sleep ?? defaultSleep; + const rps = options.requestsPerSecond ?? MIRROR_REQUESTS_PER_SECOND; + const gapMs = rps > 0 ? 1000 / rps : 0; + const maxRetries = Math.max(0, options.maxRateLimitRetries ?? MIRROR_MAX_RATE_LIMIT_RETRIES); + const secrets = [token]; + + /** The earliest moment the next request may start. */ + let nextStartAt = Number.NEGATIVE_INFINITY; + + function remainingMs(): number { + return deadline === undefined ? Number.POSITIVE_INFINITY : deadline - now(); + } + + function deadlineError(): NotionMirrorError { + return new NotionMirrorError("deadline", "Notion request not made: the time budget for this run is spent."); + } + + /** Reserve the next start slot (synchronously), then wait for it. */ + async function throttle(): Promise { + const current = now(); + const startAt = Math.max(current, nextStartAt); + if (deadline !== undefined && startAt >= deadline) throw deadlineError(); + nextStartAt = startAt + gapMs; + const wait = startAt - current; + if (wait > 0) await sleep(wait); + } + + async function request(method: string, path: string, body: unknown, notFound: NotFoundCode): Promise { + const url = `${(options.baseUrl ?? notionApiBaseUrl()).replace(/\/+$/, "")}${path}`; + for (let attempt = 0; ; attempt += 1) { + await throttle(); + const remaining = remainingMs(); + if (remaining <= 0) throw deadlineError(); + + let response: Response; + try { + response = await fetch(url, { + method, + headers: { + Authorization: `Bearer ${token}`, + "Notion-Version": NOTION_API_VERSION, + "Content-Type": "application/json", + }, + body: body === undefined ? undefined : JSON.stringify(body), + // Deliberately not the remaining budget (see the header): a started + // request runs to Notion's answer or this ceiling. + signal: AbortSignal.timeout(NOTION_REQUEST_TIMEOUT_MS), + }); + } catch (error) { + const name = (error as { name?: string } | null)?.name; + if (name === "TimeoutError" || name === "AbortError") { + throw new NotionMirrorError( + "unavailable", + `Notion did not answer within ${NOTION_REQUEST_TIMEOUT_MS / 1000} seconds.` + ); + } + throw new NotionMirrorError("unavailable", "Notion could not be reached (network error)."); + } + + if (response.ok) { + try { + return (await response.json()) as T; + } catch { + throw new NotionMirrorError("unavailable", `Notion ${response.status}: the response was not JSON.`, response.status); + } + } + + if (response.status === 429) { + const waitMs = retryAfterMs(response.headers.get("Retry-After"), now()); + await discard(response); + if (attempt >= maxRetries) { + throw new NotionMirrorError("rate_limited", "Notion 429 rate_limited: retries exhausted.", 429, "rate_limited"); + } + if (deadline !== undefined && now() + waitMs >= deadline) { + throw new NotionMirrorError( + "rate_limited", + "Notion 429 rate_limited: Retry-After runs past the time budget.", + 429, + "rate_limited" + ); + } + // Every request on this client waits, not only this one: the limit is + // the integration's, so the next request would be refused as well. + nextStartAt = Math.max(nextStartAt, now() + waitMs); + continue; + } + + throw await errorFrom(response, notFound, secrets); + } + } + + return { + getPage: (id) => request("GET", `/pages/${encodeURIComponent(id)}`, undefined, "page_not_found"), + getDatabase: (id) => request("GET", `/databases/${encodeURIComponent(id)}`, undefined, "database_not_found"), + createDatabase: (body) => request("POST", "/databases", body, "page_not_found"), + updateDatabase: (id, body) => request("PATCH", `/databases/${encodeURIComponent(id)}`, body, "database_not_found"), + createPage: (body) => request("POST", "/pages", body, "database_not_found"), + updatePage: (id, body) => request("PATCH", `/pages/${encodeURIComponent(id)}`, body, "page_not_found"), + remainingMs, + }; +} + +/** The plain text of a page's title property — whichever property has type `title`. */ +export function pageTitle(page: NotionPageObject): string | null { + for (const property of Object.values(page.properties ?? {})) { + if (property?.type !== "title") continue; + const text = (property.title ?? []).map((part) => part.plain_text ?? "").join("").trim(); + return text || null; + } + return null; +} + +const TOKEN_SHAPED = /(secret_|ntn_)[A-Za-z0-9]+/g; +const EMAIL_SHAPED = /[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}/g; + +/** + * `text` with each of `secrets` and anything shaped like a Notion token + * (`secret_…`, `ntn_…`) replaced by `[redacted]`. Email addresses are replaced + * too: this is what stands between Notion's error text and a status line an + * admin may paste into an issue, and the mirror carries reporter emails + * (2026-09-23 amendment), which must never reach a log. + */ +export function scrubSecrets(text: string, secrets: string[] = []): string { + let out = text; + for (const secret of secrets) { + if (secret) out = out.split(secret).join("[redacted]"); + } + return out.replace(TOKEN_SHAPED, "[redacted]").replace(EMAIL_SHAPED, "[email]"); +} + +async function errorFrom(response: Response, notFound: NotFoundCode, secrets: string[]): Promise { + const status = response.status; + let notionCode: string | null = null; + let message = ""; + try { + const text = await response.text(); + try { + const parsed = JSON.parse(text) as { code?: unknown; message?: unknown }; + if (typeof parsed.code === "string") notionCode = parsed.code; + if (typeof parsed.message === "string") message = parsed.message; + } catch { + message = text; + } + } catch { + // An unreadable body leaves the status to speak for itself. + } + const safeCode = notionCode ? scrubSecrets(notionCode, secrets).slice(0, 60) : null; + const detail = scrubSecrets(message, secrets).replace(/\s+/g, " ").trim().slice(0, MAX_MESSAGE_LENGTH); + const text = `Notion ${status}${safeCode ? ` ${safeCode}` : ""}${detail ? `: ${detail}` : ""}`; + return new NotionMirrorError(codeForStatus(status, notFound), text, status, safeCode); +} + +function codeForStatus(status: number, notFound: NotFoundCode): NotionErrorCode { + if (status === 401) return "unauthorized"; + if (status === 403) return "restricted"; + if (status === 404) return notFound; + if (status === 409) return "conflict"; + if (status === 429) return "rate_limited"; + if (status >= 500) return "unavailable"; + return "validation"; +} + +/** `Retry-After` as milliseconds: seconds, or an HTTP date; 1 s when absent or unreadable. */ +function retryAfterMs(header: string | null, nowMs: number): number { + if (header !== null && header.trim() !== "") { + const seconds = Number(header); + if (Number.isFinite(seconds) && seconds >= 0) return seconds * 1000; + const date = Date.parse(header); + if (Number.isFinite(date)) return Math.max(0, date - nowMs); + } + return 1000; +} + +async function discard(response: Response): Promise { + try { + // Read to the end so the connection can be reused. (Cancelling the stream + // instead never settles under some fetch interceptors.) + await response.arrayBuffer(); + } catch { + // Nothing to do: the body is not used either way. + } +} + +function defaultSleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} diff --git a/v5/src/lib/mirror/notion-id.test.ts b/v5/src/lib/mirror/notion-id.test.ts new file mode 100644 index 0000000..ea43729 --- /dev/null +++ b/v5/src/lib/mirror/notion-id.test.ts @@ -0,0 +1,56 @@ +import { parseNotionId } from "./notion-id"; + +const HEX = "0f5e4a3c11112222333344445555aaaa"; +const ID = "0f5e4a3c-1111-2222-3333-44445555aaaa"; + +describe("parseNotionId", () => { + it("accepts a bare 32-hex id, in either case", () => { + expect(parseNotionId(HEX)).toBe(ID); + expect(parseNotionId(HEX.toUpperCase())).toBe(ID); + expect(parseNotionId(` ${HEX}\n`)).toBe(ID); + }); + + it("accepts a dashed uuid and lower-cases it", () => { + expect(parseNotionId(ID)).toBe(ID); + expect(parseNotionId(ID.toUpperCase())).toBe(ID); + }); + + it("reads the id off the end of a slugged notion.so URL", () => { + expect(parseNotionId(`https://www.notion.so/MakerLab-Tools-mirror-${HEX}`)).toBe(ID); + expect(parseNotionId(`https://www.notion.so/cornell-tech/MakerLab-Tools-mirror-${HEX}`)).toBe(ID); + expect(parseNotionId(`https://notion.so/${HEX}`)).toBe(ID); + // No scheme, as copied out of some address bars. + expect(parseNotionId(`www.notion.so/Page-${HEX}`)).toBe(ID); + }); + + it("ignores ?v=, ?pvs= and the hash", () => { + const view = "9999888877776666555544443333aaaa"; + expect(parseNotionId(`https://www.notion.so/${HEX}?v=${view}`)).toBe(ID); + expect(parseNotionId(`https://www.notion.so/Tools-${HEX}?pvs=4`)).toBe(ID); + expect(parseNotionId(`https://www.notion.so/Tools-${HEX}?v=${view}&pvs=4#${view}`)).toBe(ID); + }); + + it("accepts notion.site public pages", () => { + expect(parseNotionId(`https://cornell-makerlab.notion.site/Mirror-${HEX}`)).toBe(ID); + expect(parseNotionId(`https://notion.site/${ID}`)).toBe(ID); + }); + + it("takes the last id in the path", () => { + const other = "11112222333344445555666677778888"; + expect(parseNotionId(`https://www.notion.so/${other}/Child-${HEX}`)).toBe(ID); + }); + + it("returns null for anything else", () => { + expect(parseNotionId("")).toBeNull(); + expect(parseNotionId(" ")).toBeNull(); + expect(parseNotionId("not an id")).toBeNull(); + expect(parseNotionId(HEX.slice(1))).toBeNull(); + expect(parseNotionId(`${HEX}0`)).toBeNull(); + expect(parseNotionId(`https://example.com/Page-${HEX}`)).toBeNull(); + expect(parseNotionId(`https://evilnotion.so/Page-${HEX}`)).toBeNull(); + expect(parseNotionId("https://www.notion.so/Just-a-slug")).toBeNull(); + // A 33-hex run is not an id with a stray character; it is not an id. + expect(parseNotionId(`https://www.notion.so/a${HEX}`)).toBeNull(); + expect(parseNotionId(`javascript:${HEX}`)).toBeNull(); + }); +}); diff --git a/v5/src/lib/mirror/notion-id.ts b/v5/src/lib/mirror/notion-id.ts new file mode 100644 index 0000000..9d0a93f --- /dev/null +++ b/v5/src/lib/mirror/notion-id.ts @@ -0,0 +1,70 @@ +/** + * Notion page and database ids as an admin pastes them (spec §3.8 "Connect", + * "Mapping"). + * + * People paste whatever the browser shows: a bare 32-hex id, a dashed uuid, or + * a `notion.so` / `notion.site` URL whose last path segment is a slug with the + * id on the end (`/My-Page-0f5e…`), often followed by `?v=` or + * `?pvs=4`. The id is always in the path; the query and the hash name a view or + * a block and are ignored. + * + * Returns the id as a dashed lower-case uuid — the form Notion's API returns and + * the form stored in `notion_mirrors` — or null for anything else. Pure; no + * imports, so the client and step code can both use it. + */ + +const BARE = /^[0-9a-f]{32}$/i; +const DASHED = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; +/** A 32-hex run at the end of a path segment, not glued to more hex before it. */ +const SEGMENT_TAIL = /(?:^|[^0-9a-f])([0-9a-f]{32})$/i; +const SEGMENT_DASHED = /(?:^|[^0-9a-f])([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})$/i; + +export function parseNotionId(input: string): string | null { + const value = input.trim(); + if (!value) return null; + if (BARE.test(value)) return dashed(value); + if (DASHED.test(value)) return value.toLowerCase(); + return fromUrl(value); +} + +function fromUrl(value: string): string | null { + let url: URL; + try { + url = new URL(/^[a-z][a-z0-9+.-]*:\/\//i.test(value) ? value : `https://${value}`); + } catch { + return null; + } + if (url.protocol !== "https:" && url.protocol !== "http:") return null; + if (!isNotionHost(url.hostname)) return null; + + let segments: string[]; + try { + segments = url.pathname.split("/").map((segment) => decodeURIComponent(segment)); + } catch { + return null; + } + for (let i = segments.length - 1; i >= 0; i -= 1) { + const segment = segments[i]; + if (!segment) continue; + const bare = SEGMENT_TAIL.exec(segment); + if (bare) return dashed(bare[1]); + const withDashes = SEGMENT_DASHED.exec(segment); + if (withDashes) return withDashes[1].toLowerCase(); + } + return null; +} + +function isNotionHost(hostname: string): boolean { + const host = hostname.toLowerCase(); + return ( + host === "notion.so" || + host.endsWith(".notion.so") || + host === "notion.site" || + host.endsWith(".notion.site") + ); +} + +function dashed(hex: string): string { + const h = hex.toLowerCase(); + return `${h.slice(0, 8)}-${h.slice(8, 12)}-${h.slice(12, 16)}-${h.slice(16, 20)}-${h.slice(20)}`; +} diff --git a/v5/src/lib/mirror/owner-roles.test.ts b/v5/src/lib/mirror/owner-roles.test.ts new file mode 100644 index 0000000..fa93178 --- /dev/null +++ b/v5/src/lib/mirror/owner-roles.test.ts @@ -0,0 +1,10 @@ +import { can } from "../auth/permissions"; +import { ROLES } from "../db/schema/vocabulary"; +import { MIRROR_OWNER_ROLES } from "./owner-roles"; + +describe("MIRROR_OWNER_ROLES", () => { + it("is exactly the stored roles that can() grants mirror.manage", () => { + const granted = ROLES.filter((role) => can({ role }, "mirror.manage")); + expect([...MIRROR_OWNER_ROLES].sort()).toEqual([...granted].sort()); + }); +}); diff --git a/v5/src/lib/mirror/owner-roles.ts b/v5/src/lib/mirror/owner-roles.ts new file mode 100644 index 0000000..b305477 --- /dev/null +++ b/v5/src/lib/mirror/owner-roles.ts @@ -0,0 +1,22 @@ +import type { Role } from "../db/schema/vocabulary.ts"; + +/** + * The stored roles that hold `mirror.manage` (spec §3.5, §8 "owner-only"). + * + * A mirror pushes only while its owner still holds that permission and is not + * banned: every claim in `data/mirrors.ts` joins the owner's `user` row and + * requires `role` in this list and `banned is not true`. Demoting or banning + * an admin therefore stops their mirror on the next trigger — without it, a + * demoted admin's personal Notion would keep receiving reporter names and + * emails, and nobody left in the app could pause it. + * + * A constant rather than a call to `can()`: this list is read by workflow step + * code under plain Node, which must not load Better Auth's access-control + * module. `owner-roles.test.ts` derives the same list from `can()` and fails + * the moment the two disagree. + * + * The super-admin floor (`AUTH_SUPER_ADMIN_EMAILS`) is not consulted: the row + * is what the claim reads, and a floor address's row is set right by + * `reconcileSuperAdminFloor`. + */ +export const MIRROR_OWNER_ROLES: readonly Role[] = ["admin", "super_admin"]; diff --git a/v5/src/lib/mirror/properties.test.ts b/v5/src/lib/mirror/properties.test.ts new file mode 100644 index 0000000..f3e0bbe --- /dev/null +++ b/v5/src/lib/mirror/properties.test.ts @@ -0,0 +1,326 @@ +// @vitest-environment node +import type { MirrorEntity } from "../db/schema/vocabulary"; +import { + buildMirrorProperties, + categoryProperties, + filesProp, + locationProperties, + maintenanceProperties, + optionName, + projectProperties, + relationTargets, + resourceProperties, + richText, + toolProperties, + unitProperties, + type MirrorRelations, +} from "./properties"; +import { mirrorPropertySpecs } from "./database-schemas"; +import type { + MaintenanceSourceRow, + ProjectSourceRow, + ResourceSourceRow, + ToolSourceRow, + UnitSourceRow, +} from "./source"; + +/** + * The seven property builders (spec §10 "Mirror property builders for each + * entity", as amended 2026-09-23: the mirror CARRIES reporter, assignee and + * author names and emails). + */ + +const BASE = { + revision: "2026-01-01 10:00:00.123456+00", + updatedAt: "2026-01-01T10:00:00.123Z", + pageId: null, + archive: false, +}; + +const TOOL_ID = "11111111-1111-4111-8111-111111111111"; +const CATEGORY_ID = "22222222-2222-4222-8222-222222222222"; +const LOCATION_ID = "33333333-3333-4333-8333-333333333333"; +const UNIT_ID = "44444444-4444-4444-8444-444444444444"; + +/** Every target mapped, every target mirrored as `page-`. */ +const ALL_MIRRORED: MirrorRelations = { + mapped: () => true, + pageId: (_entity, id) => `page-${id}`, +}; + +function relations(pages: Partial>, mapped: MirrorEntity[] = ["categories", "locations", "tools", "units"]): MirrorRelations { + return { + mapped: (entity) => mapped.includes(entity), + pageId: (entity, id) => (pages[entity]?.includes(id) ? `page-${id}` : null), + }; +} + +function plain(value: unknown): string { + const record = value as { title?: { text: { content: string } }[]; rich_text?: { text: { content: string } }[] }; + return (record.title ?? record.rich_text ?? []).map((item) => item.text.content).join(""); +} + +const tool: ToolSourceRow = { + ...BASE, + id: TOOL_ID, + name: "Form 4", + slug: "form-4", + description: "An SLA printer.", + categoryId: CATEGORY_ID, + locationId: LOCATION_ID, + materials: ["Resin", "Resin", "PLA, PETG"], + ppeRequired: ["Gloves"], + tags: [], + trainingRequired: true, + useRestrictions: null, + emergencyStop: "Lid switch", + notes: null, + published: false, + archived: false, + lastReviewedAt: null, + images: [ + { url: "https://blob.example.com/tools/form-4.jpg", name: "form-4.jpg" }, + { url: "https://blob.example.com/tools/side", name: null }, + ], +}; + +const maintenance: MaintenanceSourceRow = { + ...BASE, + id: "55555555-5555-4555-8555-555555555555", + title: "Resin tank leaking", + type: "issue_report", + priority: "high", + status: "in_progress", + description: "Drips under the tray.", + resolution: null, + toolId: TOOL_ID, + unitId: UNIT_ID, + toolName: "Form 4", + unitLabel: "Form 4 #1", + reportedByName: "Ada Student", + reportedByEmail: "ada@cornell.edu", + assignedToName: "Niti", + assigneeEmail: "niti@cornell.edu", + dateReported: "2026-02-01", + dateResolved: null, +}; + +const project: ProjectSourceRow = { + ...BASE, + id: "66666666-6666-4666-8666-666666666666", + title: "Plywood lamp", + link: "https://example.com/lamp", + body: "Cut on the laser.", + materials: ["Plywood"], + toolIds: [TOOL_ID], + authorName: "Luis", + authorEmail: "luis@cornell.edu", + published: true, + publishedAt: "2026-03-02T15:00:00.000Z", + photos: [{ url: "https://blob.example.com/projects/lamp.jpg", name: "lamp.jpg" }], +}; + +describe("mirror property builders", () => { + it("categories: Name, Group, App ID, Updated", () => { + const { properties, missingRelation } = categoryProperties({ ...BASE, id: CATEGORY_ID, name: "3D Printing", group: null }); + expect(missingRelation).toBe(false); + expect(plain(properties.Name)).toBe("3D Printing"); + expect(properties.Group).toEqual({ rich_text: [] }); + expect(plain(properties["App ID"])).toBe(CATEGORY_ID); + expect(properties.Updated).toEqual({ date: { start: "2026-01-01T10:00:00.123Z" } }); + }); + + it("locations: the title is Room — Zone", () => { + const { properties } = locationProperties({ ...BASE, id: LOCATION_ID, room: "Makerlab", zone: "Bench A", mapTag: null }); + expect(plain(properties.Name)).toBe("Makerlab — Bench A"); + expect(plain(properties.Room)).toBe("Makerlab"); + expect(plain(properties.Zone)).toBe("Bench A"); + expect(properties["Map tag"]).toEqual({ rich_text: [] }); + }); + + it("tools: a draft says so, relations resolve, selects lose commas and duplicates, public images only", () => { + const { properties, missingRelation } = toolProperties(tool, ALL_MIRRORED); + expect(missingRelation).toBe(false); + expect(properties.Published).toEqual({ checkbox: false }); + expect(properties.Archived).toEqual({ checkbox: false }); + expect(properties["Training required"]).toEqual({ checkbox: true }); + expect(properties.Category).toEqual({ relation: [{ id: `page-${CATEGORY_ID}` }] }); + expect(properties.Location).toEqual({ relation: [{ id: `page-${LOCATION_ID}` }] }); + expect(properties.Materials).toEqual({ multi_select: [{ name: "Resin" }, { name: "PLA PETG" }] }); + expect(properties.Tags).toEqual({ multi_select: [] }); + expect(properties["Use restrictions"]).toEqual({ rich_text: [] }); + expect(properties["Last reviewed"]).toEqual({ date: null }); + expect(properties.Images).toEqual({ + files: [ + { name: "form-4.jpg", type: "external", external: { url: "https://blob.example.com/tools/form-4.jpg" } }, + { name: "side", type: "external", external: { url: "https://blob.example.com/tools/side" } }, + ], + }); + + const published = toolProperties({ ...tool, published: true }, ALL_MIRRORED); + expect(published.properties.Published).toEqual({ checkbox: true }); + }); + + it("tools: a relation with no page yet is flagged; an unmapped target is left out; no category is an empty relation", () => { + const missing = toolProperties(tool, relations({ locations: [LOCATION_ID] })); + expect(missing.missingRelation).toBe(true); + expect(missing.properties.Category).toEqual({ relation: [] }); + + const unmapped = toolProperties(tool, relations({ locations: [LOCATION_ID] }, ["locations"])); + expect(unmapped.missingRelation).toBe(false); + expect("Category" in unmapped.properties).toBe(false); + + const none = toolProperties({ ...tool, categoryId: null }, relations({ locations: [LOCATION_ID] })); + expect(none.missingRelation).toBe(false); + expect(none.properties.Category).toEqual({ relation: [] }); + }); + + it("units: selects carry the stored machine ids", () => { + const unit: UnitSourceRow = { + ...BASE, + id: UNIT_ID, + toolId: TOOL_ID, + unitLabel: "Form 4 #1", + serialNumber: "SN-1", + assetTag: null, + status: "in_use", + condition: null, + dateAcquired: "2025-09-01", + notes: null, + }; + const { properties } = unitProperties(unit, ALL_MIRRORED); + expect(plain(properties.Label)).toBe("Form 4 #1"); + expect(properties.Status).toEqual({ select: { name: "in_use" } }); + expect(properties.Condition).toEqual({ select: null }); + expect(properties["Date acquired"]).toEqual({ date: { start: "2025-09-01" } }); + expect(properties.Tool).toEqual({ relation: [{ id: `page-${TOOL_ID}` }] }); + + const orphaned = unitProperties({ ...unit, toolId: null }, ALL_MIRRORED); + expect(orphaned.properties.Tool).toEqual({ relation: [] }); + }); + + it("resources: Published checkbox, URL, public files", () => { + const resource: ResourceSourceRow = { + ...BASE, + id: "77777777-7777-4777-8777-777777777777", + toolId: TOOL_ID, + title: "Manual", + type: "manual", + url: "https://formlabs.com/manual", + published: true, + notes: null, + files: [{ url: "https://blob.example.com/manual.pdf", name: "manual.pdf" }], + }; + const { properties } = resourceProperties(resource, ALL_MIRRORED); + expect(properties.Published).toEqual({ checkbox: true }); + expect(properties.URL).toEqual({ url: "https://formlabs.com/manual" }); + expect(properties.Type).toEqual({ select: { name: "manual" } }); + expect(properties.File).toEqual({ + files: [{ name: "manual.pdf", type: "external", external: { url: "https://blob.example.com/manual.pdf" } }], + }); + + const hidden = resourceProperties({ ...resource, published: false, url: `https://x.example/${"a".repeat(2001)}` }, ALL_MIRRORED); + expect(hidden.properties.Published).toEqual({ checkbox: false }); + expect(hidden.properties.URL).toEqual({ url: null }); + }); + + it("maintenance: CARRIES the reporter's and the assignee's names and emails (2026-09-23 amendment)", () => { + const { properties, missingRelation } = maintenanceProperties(maintenance, ALL_MIRRORED); + expect(missingRelation).toBe(false); + expect(plain(properties["Reported by"])).toBe("Ada Student"); + expect(properties["Reporter email"]).toEqual({ email: "ada@cornell.edu" }); + expect(plain(properties["Assigned to"])).toBe("Niti"); + expect(properties["Assignee email"]).toEqual({ email: "niti@cornell.edu" }); + expect(properties.Type).toEqual({ select: { name: "issue_report" } }); + expect(properties.Status).toEqual({ select: { name: "in_progress" } }); + expect(properties.Tool).toEqual({ relation: [{ id: `page-${TOOL_ID}` }] }); + expect(properties.Unit).toEqual({ relation: [{ id: `page-${UNIT_ID}` }] }); + expect(properties["Date resolved"]).toEqual({ date: null }); + + const anonymous = maintenanceProperties( + { ...maintenance, reportedByEmail: null, assigneeEmail: " ", reportedByName: null }, + ALL_MIRRORED + ); + expect(anonymous.properties["Reporter email"]).toEqual({ email: null }); + expect(anonymous.properties["Assignee email"]).toEqual({ email: null }); + expect(anonymous.properties["Reported by"]).toEqual({ rich_text: [] }); + }); + + it("projects: CARRIES the author's name and email, relates every tool, public photos", () => { + const { properties } = projectProperties(project, ALL_MIRRORED); + expect(plain(properties.Author)).toBe("Luis"); + expect(properties["Author email"]).toEqual({ email: "luis@cornell.edu" }); + expect(properties.Tools).toEqual({ relation: [{ id: `page-${TOOL_ID}` }] }); + expect(properties.Link).toEqual({ url: "https://example.com/lamp" }); + expect(properties["Published at"]).toEqual({ date: { start: "2026-03-02T15:00:00.000Z" } }); + expect(properties.Photos).toEqual({ + files: [{ name: "lamp.jpg", type: "external", external: { url: "https://blob.example.com/projects/lamp.jpg" } }], + }); + + const partly = projectProperties({ ...project, toolIds: [TOOL_ID, UNIT_ID] }, relations({ tools: [TOOL_ID] })); + expect(partly.missingRelation).toBe(true); + expect(partly.properties.Tools).toEqual({ relation: [{ id: `page-${TOOL_ID}` }] }); + }); + + it("writes only properties the entity's schema declares", () => { + const rows: Record = { + categories: { ...BASE, id: CATEGORY_ID, name: "x", group: "y" }, + locations: { ...BASE, id: LOCATION_ID, room: "r", zone: "z", mapTag: "m" }, + tools: tool, + units: { ...BASE, id: UNIT_ID, toolId: TOOL_ID, unitLabel: "u", serialNumber: null, assetTag: null, status: "available", condition: "good", dateAcquired: null, notes: null }, + resources: { ...BASE, id: TOOL_ID, toolId: TOOL_ID, title: "t", type: null, url: null, published: true, notes: null, files: [] }, + maintenance, + projects: project, + }; + for (const [entity, row] of Object.entries(rows) as [MirrorEntity, never][]) { + const declared = new Set(mirrorPropertySpecs(entity).map((spec) => spec.name)); + const built = Object.keys(buildMirrorProperties(entity, row, ALL_MIRRORED).properties); + expect(built.filter((name) => !declared.has(name)), entity).toEqual([]); + // Every declared property is written, so clearing a field in the app clears it in Notion. + expect([...declared].filter((name) => !built.includes(name)), entity).toEqual([]); + } + }); +}); + +describe("Notion limits", () => { + it("chunks long text into 2000-character items, at most 100", () => { + const long = "a".repeat(4500); + const items = richText(long); + expect(items.map((item) => item.text.content.length)).toEqual([2000, 2000, 500]); + expect(richText("b".repeat(2000 * 150))).toHaveLength(100); + expect(richText("")).toEqual([]); + expect(richText(null)).toEqual([]); + }); + + it("never splits a surrogate pair across two items", () => { + const text = `${"a".repeat(1999)}😀tail`; + const items = richText(text); + expect(items[0].text.content).toBe("a".repeat(1999)); + expect(items[1].text.content).toBe("😀tail"); + }); + + it("strips commas from option names, trims, and caps them at 100 characters", () => { + expect(optionName("PLA, PETG")).toBe("PLA PETG"); + expect(optionName(" , ")).toBeNull(); + expect(optionName("x".repeat(150))).toHaveLength(100); + expect(optionName(null)).toBeNull(); + }); + + it("names files, capping the name at 100 characters and skipping over-long URLs", () => { + const value = filesProp( + [ + { url: `https://blob.example.com/${"n".repeat(150)}`, name: null }, + { url: `https://blob.example.com/${"u".repeat(2001)}`, name: "too-long" }, + ], + "Image" + ) as { files: { name: string }[] }; + expect(value.files).toHaveLength(1); + expect(value.files[0].name).toHaveLength(100); + }); + + it("collects the relation targets a batch needs", () => { + const targets = relationTargets("maintenance", [maintenance, { ...maintenance, toolId: null, unitId: null }]); + expect([...targets.keys()].sort()).toEqual(["tools", "units"]); + expect([...targets.get("tools")!]).toEqual([TOOL_ID]); + }); +}); diff --git a/v5/src/lib/mirror/properties.ts b/v5/src/lib/mirror/properties.ts new file mode 100644 index 0000000..5559cf3 --- /dev/null +++ b/v5/src/lib/mirror/properties.ts @@ -0,0 +1,375 @@ +import type { MirrorEntity } from "../db/schema/vocabulary.ts"; +import { APP_ID_PROPERTY, UPDATED_PROPERTY } from "./database-schemas.ts"; +import type { + AnySourceRow, + CategorySourceRow, + LocationSourceRow, + MaintenanceSourceRow, + MirrorFile, + ProjectSourceRow, + ResourceSourceRow, + SourceRows, + ToolSourceRow, + UnitSourceRow, +} from "./source.ts"; + +/** + * Pure builders: one source row → the Notion `properties` of its page (spec + * §3.8; the schemas are in `database-schemas.ts`). + * + * They respect Notion's request limits, so a long description or a stray + * comma cannot fail a push: + * + * - text is split into rich-text items of at most 2000 characters, at most + * 100 of them (never cutting a surrogate pair in half); + * - select and multi-select names lose their commas (Notion refuses them), are + * cut to 100 characters, and are de-duplicated; empty names are dropped; + * - a URL longer than 2000 characters is left out rather than truncated into + * a different URL; + * - files are external `{ name, type: "external", external: { url } }` items, + * names cut to 100 characters; + * - every empty value is sent explicitly (`[]`, `null`, `false`), so clearing + * a field in the app clears it in Notion. + * + * **Emails are carried** (2026-09-23 amendment): the maintenance builder sends + * the reporter's and the assignee's name and email, the project builder the + * author's. Nothing here logs. + * + * **Relations** are resolved through a {@link MirrorRelations} the push builds + * from `mirror_pages`. A target whose entity is not mapped leaves the property + * out (there is no database to point at). A target that should have a page + * and does not yet sets `missingRelation`, so the push records the row as not + * yet mirrored and tries it again next time. + */ + +export const NOTION_TEXT_CHUNK = 2000; +export const NOTION_MAX_RICH_TEXT_ITEMS = 100; +export const NOTION_MAX_OPTION_LENGTH = 100; +export const NOTION_MAX_URL_LENGTH = 2000; +export const NOTION_MAX_FILE_NAME_LENGTH = 100; +export const NOTION_MAX_ARRAY_ITEMS = 100; + +/** Page ids of already-mirrored rows, by target entity. */ +export interface MirrorRelations { + /** Whether `entity` has a database in this mirror's mapping. */ + mapped(entity: MirrorEntity): boolean; + /** The Notion page mirroring `entity` row `id`, or null when it has none yet. */ + pageId(entity: MirrorEntity, id: string): string | null; +} + +export interface BuiltProperties { + properties: Record; + /** A relation target that should exist in Notion does not yet. */ + missingRelation: boolean; +} + +type Properties = Record; + +// ── Value helpers ─────────────────────────────────────────────────── + +/** Text as rich-text items: ≤2000 characters each, ≤100 items, `[]` when empty. */ +export function richText(value: string | null | undefined): { type: "text"; text: { content: string } }[] { + const text = value ?? ""; + if (text === "") return []; + const items: { type: "text"; text: { content: string } }[] = []; + let start = 0; + while (start < text.length && items.length < NOTION_MAX_RICH_TEXT_ITEMS) { + let end = Math.min(text.length, start + NOTION_TEXT_CHUNK); + // Never split a surrogate pair across two items. + if (end < text.length && isHighSurrogate(text.charCodeAt(end - 1))) end -= 1; + items.push({ type: "text", text: { content: text.slice(start, end) } }); + start = end; + } + return items; +} + +function isHighSurrogate(code: number): boolean { + return code >= 0xd800 && code <= 0xdbff; +} + +/** A select / multi-select option name: no commas, trimmed, ≤100 characters; null when nothing is left. */ +export function optionName(value: string | null | undefined): string | null { + if (typeof value !== "string") return null; + const cleaned = value.replace(/,/g, "").replace(/\s+/g, " ").trim(); + if (!cleaned) return null; + return truncate(cleaned, NOTION_MAX_OPTION_LENGTH).trim() || null; +} + +function truncate(value: string, max: number): string { + if (value.length <= max) return value; + let end = max; + if (isHighSurrogate(value.charCodeAt(end - 1))) end -= 1; + return value.slice(0, end); +} + +function titleProp(value: string | null | undefined): Properties[string] { + return { title: richText(value) }; +} + +function textProp(value: string | null | undefined): Properties[string] { + return { rich_text: richText(value) }; +} + +function selectProp(value: string | null | undefined): Properties[string] { + const name = optionName(value); + return { select: name ? { name } : null }; +} + +function multiSelectProp(values: readonly string[]): Properties[string] { + const seen = new Set(); + const out: { name: string }[] = []; + for (const value of values) { + const name = optionName(value); + if (!name || seen.has(name)) continue; + seen.add(name); + out.push({ name }); + if (out.length >= NOTION_MAX_ARRAY_ITEMS) break; + } + return { multi_select: out }; +} + +function checkboxProp(value: boolean): Properties[string] { + return { checkbox: value === true }; +} + +function dateProp(value: string | null | undefined): Properties[string] { + return { date: value ? { start: value } : null }; +} + +/** A URL, trimmed; null when empty or longer than Notion takes. */ +export function urlValue(value: string | null | undefined): string | null { + const url = (value ?? "").trim(); + if (!url || url.length > NOTION_MAX_URL_LENGTH) return null; + return url; +} + +function urlProp(value: string | null | undefined): Properties[string] { + return { url: urlValue(value) }; +} + +function emailProp(value: string | null | undefined): Properties[string] { + const email = (value ?? "").trim(); + return { email: email || null }; +} + +/** External files: public URLs only reach here (source.ts selects nothing else). */ +export function filesProp(files: readonly MirrorFile[], fallbackName: string): Properties[string] { + const out: { name: string; type: "external"; external: { url: string } }[] = []; + for (const file of files) { + const url = urlValue(file.url); + if (!url) continue; + const name = truncate((file.name?.trim() || nameFromUrl(url) || `${fallbackName} ${out.length + 1}`).trim(), NOTION_MAX_FILE_NAME_LENGTH); + out.push({ name: name || `${fallbackName} ${out.length + 1}`, type: "external", external: { url } }); + if (out.length >= NOTION_MAX_ARRAY_ITEMS) break; + } + return { files: out }; +} + +function nameFromUrl(url: string): string | null { + try { + const last = new URL(url).pathname.split("/").filter(Boolean).pop(); + return last ? decodeURIComponent(last) : null; + } catch { + return null; + } +} + +/** Relation state collected while a row's properties are built. */ +class RelationSink { + missing = false; + private readonly relations: MirrorRelations; + + constructor(relations: MirrorRelations) { + this.relations = relations; + } + + /** Set `properties[name]` to the relation, or leave it out when the target is not mapped. */ + one(properties: Properties, name: string, target: MirrorEntity, id: string | null): void { + this.many(properties, name, target, id ? [id] : []); + } + + many(properties: Properties, name: string, target: MirrorEntity, ids: readonly string[]): void { + if (!this.relations.mapped(target)) return; + const out: { id: string }[] = []; + for (const id of new Set(ids)) { + const pageId = this.relations.pageId(target, id); + if (pageId) out.push({ id: pageId }); + else this.missing = true; + if (out.length >= NOTION_MAX_ARRAY_ITEMS) break; + } + properties[name] = { relation: out }; + } +} + +function common(row: AnySourceRow): Properties { + return { [APP_ID_PROPERTY]: textProp(row.id), [UPDATED_PROPERTY]: dateProp(row.updatedAt) }; +} + +// ── Builders ──────────────────────────────────────────────────────── + +export function categoryProperties(row: CategorySourceRow): BuiltProperties { + return { + properties: { Name: titleProp(row.name), Group: textProp(row.group), ...common(row) }, + missingRelation: false, + }; +} + +export function locationProperties(row: LocationSourceRow): BuiltProperties { + return { + properties: { + Name: titleProp(`${row.room} — ${row.zone}`), + Room: textProp(row.room), + Zone: textProp(row.zone), + "Map tag": textProp(row.mapTag), + ...common(row), + }, + missingRelation: false, + }; +} + +export function toolProperties(row: ToolSourceRow, relations: MirrorRelations): BuiltProperties { + const sink = new RelationSink(relations); + const properties: Properties = { + Name: titleProp(row.name), + Slug: textProp(row.slug), + Description: textProp(row.description), + Materials: multiSelectProp(row.materials), + "PPE required": multiSelectProp(row.ppeRequired), + Tags: multiSelectProp(row.tags), + "Training required": checkboxProp(row.trainingRequired), + "Use restrictions": textProp(row.useRestrictions), + "Emergency stop": textProp(row.emergencyStop), + Notes: textProp(row.notes), + Published: checkboxProp(row.published), + Archived: checkboxProp(row.archived), + Images: filesProp(row.images, "Image"), + "Last reviewed": dateProp(row.lastReviewedAt), + ...common(row), + }; + sink.one(properties, "Category", "categories", row.categoryId); + sink.one(properties, "Location", "locations", row.locationId); + return { properties, missingRelation: sink.missing }; +} + +export function unitProperties(row: UnitSourceRow, relations: MirrorRelations): BuiltProperties { + const sink = new RelationSink(relations); + const properties: Properties = { + Label: titleProp(row.unitLabel), + "Serial number": textProp(row.serialNumber), + "Asset tag": textProp(row.assetTag), + Status: selectProp(row.status), + Condition: selectProp(row.condition), + "Date acquired": dateProp(row.dateAcquired), + Notes: textProp(row.notes), + ...common(row), + }; + sink.one(properties, "Tool", "tools", row.toolId); + return { properties, missingRelation: sink.missing }; +} + +export function resourceProperties(row: ResourceSourceRow, relations: MirrorRelations): BuiltProperties { + const sink = new RelationSink(relations); + const properties: Properties = { + Title: titleProp(row.title), + Type: selectProp(row.type), + URL: urlProp(row.url), + File: filesProp(row.files, "File"), + Published: checkboxProp(row.published), + Notes: textProp(row.notes), + ...common(row), + }; + sink.one(properties, "Tool", "tools", row.toolId); + return { properties, missingRelation: sink.missing }; +} + +export function maintenanceProperties(row: MaintenanceSourceRow, relations: MirrorRelations): BuiltProperties { + const sink = new RelationSink(relations); + const properties: Properties = { + Title: titleProp(row.title), + Type: selectProp(row.type), + Priority: selectProp(row.priority), + Status: selectProp(row.status), + Description: textProp(row.description), + Resolution: textProp(row.resolution), + "Tool name": textProp(row.toolName), + "Unit label": textProp(row.unitLabel), + "Reported by": textProp(row.reportedByName), + "Reporter email": emailProp(row.reportedByEmail), + "Assigned to": textProp(row.assignedToName), + "Assignee email": emailProp(row.assigneeEmail), + "Date reported": dateProp(row.dateReported), + "Date resolved": dateProp(row.dateResolved), + ...common(row), + }; + sink.one(properties, "Tool", "tools", row.toolId); + sink.one(properties, "Unit", "units", row.unitId); + return { properties, missingRelation: sink.missing }; +} + +export function projectProperties(row: ProjectSourceRow, relations: MirrorRelations): BuiltProperties { + const sink = new RelationSink(relations); + const properties: Properties = { + Title: titleProp(row.title), + Link: urlProp(row.link), + Body: textProp(row.body), + Materials: multiSelectProp(row.materials), + Author: textProp(row.authorName), + "Author email": emailProp(row.authorEmail), + "Published at": dateProp(row.publishedAt), + Photos: filesProp(row.photos, "Photo"), + ...common(row), + }; + sink.many(properties, "Tools", "tools", row.toolIds); + return { properties, missingRelation: sink.missing }; +} + +/** The builder for `entity`. */ +export function buildMirrorProperties( + entity: E, + row: SourceRows[E], + relations: MirrorRelations +): BuiltProperties { + switch (entity) { + case "categories": + return categoryProperties(row as CategorySourceRow); + case "locations": + return locationProperties(row as LocationSourceRow); + case "tools": + return toolProperties(row as ToolSourceRow, relations); + case "units": + return unitProperties(row as UnitSourceRow, relations); + case "resources": + return resourceProperties(row as ResourceSourceRow, relations); + case "maintenance": + return maintenanceProperties(row as MaintenanceSourceRow, relations); + case "projects": + return projectProperties(row as ProjectSourceRow, relations); + default: + return { properties: {}, missingRelation: false }; + } +} + +/** Which ids of which target entities `rows` relate to — what the push looks up in `mirror_pages`. */ +export function relationTargets(entity: E, rows: readonly SourceRows[E][]): Map> { + const out = new Map>(); + const add = (target: MirrorEntity, id: string | null) => { + if (!id) return; + let set = out.get(target); + if (!set) out.set(target, (set = new Set())); + set.add(id); + }; + for (const row of rows as readonly AnySourceRow[]) { + if (entity === "tools") { + add("categories", (row as ToolSourceRow).categoryId); + add("locations", (row as ToolSourceRow).locationId); + } else if (entity === "units" || entity === "resources") { + add("tools", (row as UnitSourceRow | ResourceSourceRow).toolId); + } else if (entity === "maintenance") { + add("tools", (row as MaintenanceSourceRow).toolId); + add("units", (row as MaintenanceSourceRow).unitId); + } else if (entity === "projects") { + for (const id of (row as ProjectSourceRow).toolIds) add("tools", id); + } + } + return out; +} diff --git a/v5/src/lib/mirror/push.test.ts b/v5/src/lib/mirror/push.test.ts new file mode 100644 index 0000000..5a9a13b --- /dev/null +++ b/v5/src/lib/mirror/push.test.ts @@ -0,0 +1,506 @@ +// @vitest-environment node +import { eq, sql } from "drizzle-orm"; +import { server } from "../../../test/msw/server"; +// Aliased: the name starts with `use`, which eslint's rules-of-hooks reads as a React hook. +import { useNotionFake as installNotionFake } from "../../../test/msw/notion-mirror"; +import { createNotionFake, type NotionFake, type NotionFakeResponse } from "../../../test/fakes/notion-fake"; +import { seedDemo } from "../db/demo-seed"; +import { createPgliteDb } from "../db/pglite"; +import { rawRows } from "../db/raw"; +import { categories, maintenanceLogs, mirrorPages, notionMirrors, projects, tools, units, user } from "../db/schema/index"; +import { MIRROR_ENTITY, type MirrorEntity } from "../db/schema/vocabulary"; +import type { Db } from "../db/types"; +import { getMirror, resetMirrorEntities, saveMirrorConnection, setMirrorPaused } from "../data/mirrors"; +import { ensureMirrorDatabases } from "./databases"; +import { pushMirror, type MirrorPushOutcome } from "./push"; +import { encryptMirrorToken } from "./token-crypto"; +import type { MirrorMapping } from "./types"; + +/** + * The push against the fake Notion (spec §10 "Mirror against mocked Notion", + * §3.8, §5.8). Every test gets its own seeded PGlite and its own fake, and + * drives the client with a fake clock: `sleep` advances it, so the throttle, + * `Retry-After` and the budget are all exact and instant. + */ + +const TOKEN = "ntn_PUSHtoken0123456789abcdefSECRET"; +const SECRET = "test-auth-secret-for-mirror-push"; +const PARENT = "0f5e4a3c-1111-2222-3333-44445555aaaa"; + +interface Harness { + db: Db; + fake: NotionFake; + mirrorId: string; + ownerId: string; + mapping: MirrorMapping; + clock: { t: number }; + sleep: ReturnType Promise>>; + push: (options?: { budgetMs?: number; requestsPerSecond?: number }) => Promise; +} + +async function harness(options: { databases?: boolean } = {}): Promise { + vi.stubEnv("AUTH_SECRET", SECRET); + const db = await createPgliteDb({ seed: seedDemo }); + const fake = createNotionFake({ token: TOKEN, pages: [{ id: PARENT, title: "MakerLab Tools — mirror" }] }); + installNotionFake(server, fake); + + const ownerId = `owner-${crypto.randomUUID()}`; + await db.insert(user).values({ id: ownerId, name: "Mirror Owner", email: `${ownerId}@cornell.edu`, role: "admin" }); + const { mirror } = await saveMirrorConnection( + { ownerUserId: ownerId, tokenCiphertext: encryptMirrorToken(TOKEN, SECRET), parentPageId: PARENT, parentPageTitle: "Mirror" }, + { db } + ); + + const clock = { t: 1_700_000_000_000 }; + const sleep = vi.fn(async (ms: number) => { + clock.t += ms; + }); + const client = (requestsPerSecond = 0) => ({ requestsPerSecond, now: () => clock.t, sleep }); + + let mapping: MirrorMapping = {}; + if (options.databases !== false) { + const ensured = await ensureMirrorDatabases(mirror.id, { db, client: client() }); + if (!ensured.ok) throw new Error(`setup failed: ${ensured.code}`); + mapping = ensured.mapping; + fake.requests.length = 0; + } + + return { + db, + fake, + mirrorId: mirror.id, + ownerId, + mapping, + clock, + sleep, + push: (opts = {}) => + pushMirror(mirror.id, { db, budgetMs: opts.budgetMs, client: client(opts.requestsPerSecond ?? 0) }), + }; +} + +/** How many rows each entity should mirror, counted by SQL. */ +async function expectedCounts(db: Db): Promise> { + const [row] = await rawRows>( + db, + sql`select + (select count(*) from categories) as categories, + (select count(*) from locations) as locations, + (select count(*) from tools where archived_at is null) as tools, + (select count(*) from units) as units, + (select count(*) from resources) as resources, + (select count(*) from maintenance_logs) as maintenance, + (select count(*) from projects where published) as projects` + ); + return Object.fromEntries(MIRROR_ENTITY.map((entity) => [entity, Number(row[entity])])) as Record; +} + +function livePages(h: Harness, entity: MirrorEntity) { + return h.fake.pagesIn(h.mapping[entity]!).filter((page) => !page.archived); +} + +function entityOfDatabase(h: Harness, databaseId: string): MirrorEntity | undefined { + return MIRROR_ENTITY.find((entity) => h.mapping[entity] === databaseId); +} + +/** The entity each page create in the request log went to, in order. */ +function createdEntities(h: Harness): (MirrorEntity | undefined)[] { + return h.fake.requests + .filter((request) => request.method === "POST" && request.path === "/pages") + .map((request) => entityOfDatabase(h, (request.body as { parent: { database_id: string } }).parent.database_id)); +} + +function count(h: Harness, method: string, path: RegExp): number { + return h.fake.requests.filter((request) => request.method === method && path.test(request.path)).length; +} + +/** Fail page creates into one database with `status`, `times` times. */ +function failCreatesIn(h: Harness, databaseId: string, status: number, times = 1): void { + const original = h.fake.handle; + let left = times; + h.fake.handle = (method, path, headers, body) => { + const parsed = typeof body === "string" && body ? (JSON.parse(body) as { parent?: { database_id?: string } }) : null; + if (left > 0 && method === "POST" && /\/pages$/.test(path) && parsed?.parent?.database_id === databaseId) { + left -= 1; + h.fake.requests.push({ method, path: "/pages", body: parsed, at: Date.now() }); + const response: NotionFakeResponse = { + status, + headers: { "Content-Type": "application/json" }, + body: { object: "error", status, code: "internal_server_error", message: "Injected." }, + }; + return response; + } + return original(method, path, headers, body); + }; +} + +async function mirrorRow(h: Harness) { + const [row] = await rawRows<{ + last_synced_at: string | null; + last_status: string | null; + last_error: unknown; + running_since: string | null; + paused_at: string | null; + }>( + h.db, + sql`select last_synced_at::text, last_status, last_error, running_since::text, paused_at::text from notion_mirrors where id = ${h.mirrorId}` + ); + return row; +} + +describe("pushMirror", () => { + it("creates every page on the first run, and updates in place on the second", async () => { + const h = await harness(); + const expected = await expectedCounts(h.db); + + const first = await h.push(); + const total = Object.values(expected).reduce((a, b) => a + b, 0); + expect(first).toEqual({ state: "ok", pushed: total, archived: 0 }); + for (const entity of MIRROR_ENTITY) expect(livePages(h, entity), entity).toHaveLength(expected[entity]); + const afterFirst = await mirrorRow(h); + expect(afterFirst.last_status).toBe("ok"); + expect(afterFirst.last_synced_at).not.toBeNull(); + expect(afterFirst.running_since).toBeNull(); + + // Relations point at the pages the mirror made. + const [form4] = await h.db.select({ id: tools.id }).from(tools).where(eq(tools.slug, "form-4")); + const toolPage = livePages(h, "tools").find((page) => JSON.stringify(page.properties["App ID"]).includes(form4?.id ?? "none")); + expect(toolPage).toBeDefined(); + const unitPages = livePages(h, "units").filter((page) => JSON.stringify(page.properties.Tool).includes(toolPage!.id)); + expect(unitPages.length).toBeGreaterThan(0); + + h.fake.requests.length = 0; + await h.db.update(tools).set({ description: "Updated by the second run." }).where(eq(tools.id, form4.id)); + const second = await h.push(); + expect(second).toEqual({ state: "ok", pushed: 1, archived: 0 }); + expect(count(h, "POST", /^\/pages$/)).toBe(0); + expect(count(h, "PATCH", /^\/pages\//)).toBe(1); + expect(JSON.stringify(h.fake.pages.get(toolPage!.id)!.properties.Description)).toContain("Updated by the second run."); + for (const entity of MIRROR_ENTITY) expect(livePages(h, entity), entity).toHaveLength(expected[entity]); + + // A third run with nothing changed makes no request at all. + h.fake.requests.length = 0; + expect(await h.push()).toEqual({ state: "ok", pushed: 0, archived: 0 }); + expect(h.fake.requests).toEqual([]); + }); + + it("pushes in dependency order", async () => { + const h = await harness(); + await h.push(); + const order = createdEntities(h).map((entity) => MIRROR_ENTITY.indexOf(entity!)); + expect(order.length).toBeGreaterThan(0); + expect(order).toEqual([...order].sort((a, b) => a - b)); + expect(new Set(createdEntities(h))).toEqual(new Set(MIRROR_ENTITY)); + }); + + it("carries the maintenance reporter's email and the draft flag to Notion", async () => { + const h = await harness(); + const [log] = await h.db.select({ email: maintenanceLogs.reportedByEmail }).from(maintenanceLogs).where(sql`${maintenanceLogs.reportedByEmail} is not null`); + await h.db.update(tools).set({ published: false }).where(eq(tools.slug, "form-4")); + await h.push(); + expect(JSON.stringify(livePages(h, "maintenance").map((page) => page.properties["Reporter email"]))).toContain(log.email!); + const drafts = livePages(h, "tools").filter((page) => page.properties.Published.checkbox === false); + expect(drafts).toHaveLength(1); + }); + + it("archives the page of an archived tool", async () => { + const h = await harness(); + await h.push(); + const [form4] = await h.db.select({ id: tools.id }).from(tools).where(eq(tools.slug, "form-4")); + const [page] = await h.db.select({ id: mirrorPages.notionPageId }).from(mirrorPages).where(eq(mirrorPages.entityId, form4.id)); + + h.fake.requests.length = 0; + await h.db.update(tools).set({ archivedAt: new Date() }).where(eq(tools.id, form4.id)); + expect(await h.push()).toEqual({ state: "ok", pushed: 0, archived: 1 }); + expect(h.fake.pages.get(page.id)!.archived).toBe(true); + expect(h.fake.requests.find((request) => request.method === "PATCH")?.body).toEqual({ archived: true }); + + // Archived once is archived: the next push sends nothing for it. + h.fake.requests.length = 0; + expect(await h.push()).toEqual({ state: "ok", pushed: 0, archived: 0 }); + + // Unarchived, the same page comes back. + await h.db.update(tools).set({ archivedAt: null }).where(eq(tools.id, form4.id)); + expect(await h.push()).toEqual({ state: "ok", pushed: 1, archived: 0 }); + expect(h.fake.pages.get(page.id)!.archived).toBe(false); + }); + + it("archives a withdrawn project's page, and never mirrors an unpublished one", async () => { + const h = await harness(); + await h.push(); + const before = livePages(h, "projects").length; + const [published] = await h.db.select({ id: projects.id }).from(projects).where(eq(projects.published, true)).limit(1); + await h.db.update(projects).set({ published: false }).where(eq(projects.id, published.id)); + expect(await h.push()).toEqual({ state: "ok", pushed: 0, archived: 1 }); + expect(livePages(h, "projects")).toHaveLength(before - 1); + }); + + it("honours Retry-After on a 429", async () => { + const h = await harness(); + h.fake.failNext({ method: "POST", path: /^\/pages$/ }, { status: 429, retryAfter: 2 }, 1); + const outcome = await h.push(); + expect(outcome.state).toBe("ok"); + expect(h.sleep).toHaveBeenCalledWith(2000); + const expected = await expectedCounts(h.db); + for (const entity of MIRROR_ENTITY) expect(livePages(h, entity), entity).toHaveLength(expected[entity]); + }); + + it("is skipped, with no request, while another push holds the guard", async () => { + const h = await harness(); + await h.db.update(notionMirrors).set({ runningSince: sql`now()` }).where(eq(notionMirrors.id, h.mirrorId)); + expect(await h.push()).toEqual({ state: "skipped", reason: "running" }); + expect(h.fake.requests).toEqual([]); + + // A guard older than fifteen minutes belongs to a push that died. + await h.db + .update(notionMirrors) + .set({ runningSince: sql`now() - interval '16 minutes'` }) + .where(eq(notionMirrors.id, h.mirrorId)); + expect((await h.push()).state).toBe("ok"); + }); + + it("is skipped, with no request, when paused", async () => { + const h = await harness(); + await setMirrorPaused(h.ownerId, true, { db: h.db }); + expect(await h.push()).toEqual({ state: "skipped", reason: "paused" }); + expect(h.fake.requests).toEqual([]); + }); + + it("is skipped as not_mapped with no mapping, and frees the guard", async () => { + const h = await harness({ databases: false }); + expect(await h.push()).toEqual({ state: "skipped", reason: "not_mapped" }); + expect(h.fake.requests).toEqual([]); + expect((await mirrorRow(h)).running_since).toBeNull(); + }); + + it("on a partial failure keeps last_synced_at, and the next push creates only the failed row", async () => { + const h = await harness(); + await h.push(); + const synced = (await mirrorRow(h)).last_synced_at!; + + // Two changes: a new project (a leaf) whose create fails, and a renamed category that succeeds. + await h.db.insert(projects).values({ slug: `new-${crypto.randomUUID()}`, title: "New project", published: true }); + const [category] = await h.db.select({ id: categories.id }).from(categories).limit(1); + await h.db.update(categories).set({ name: "Renamed" }).where(eq(categories.id, category.id)); + failCreatesIn(h, h.mapping.projects!, 500); + + h.fake.requests.length = 0; + const partial = await h.push(); + expect(partial).toMatchObject({ state: "partial", pushed: 1, failed: 1, error: { code: "rows_failed", entities: ["projects"], failed: 1 } }); + const row = await mirrorRow(h); + expect(row.last_status).toBe("partial"); + const [same] = await rawRows<{ same: boolean }>( + h.db, + sql`select (last_synced_at = ${synced}::timestamptz) as same from notion_mirrors where id = ${h.mirrorId}` + ); + expect(same.same).toBe(true); + + h.fake.requests.length = 0; + expect(await h.push()).toEqual({ state: "ok", pushed: 1, archived: 0 }); + expect(createdEntities(h)).toEqual(["projects"]); + expect(count(h, "PATCH", /^\/pages\//)).toBe(0); + expect((await mirrorRow(h)).last_status).toBe("ok"); + }); + + it("pauses the mirror and records unauthorized on a 401", async () => { + const h = await harness(); + h.fake.failNext({}, { status: 401, code: "unauthorized" }, 1); + const outcome = await h.push(); + expect(outcome).toMatchObject({ state: "failed", paused: true, error: { code: "unauthorized" } }); + const row = await mirrorRow(h); + expect(row.paused_at).not.toBeNull(); + expect(row.last_status).toBe("failed"); + expect(row.last_error).toMatchObject({ code: "unauthorized" }); + expect(row.running_since).toBeNull(); + + h.fake.requests.length = 0; + expect(await h.push()).toEqual({ state: "skipped", reason: "paused" }); + expect(h.fake.requests).toEqual([]); + }); + + it("fails without pausing when AUTH_SECRET is unset, and pauses when the token cannot be read", async () => { + const h = await harness(); + vi.stubEnv("AUTH_SECRET", ""); + expect(await h.push()).toMatchObject({ state: "failed", paused: false, error: { code: "key_unavailable" } }); + expect((await mirrorRow(h)).paused_at).toBeNull(); + + vi.stubEnv("AUTH_SECRET", "a-rotated-auth-secret"); + expect(await h.push()).toMatchObject({ state: "failed", paused: true, error: { code: "token_unreadable" } }); + expect((await mirrorRow(h)).paused_at).not.toBeNull(); + expect(h.fake.requests).toEqual([]); + }); + + it("stops at the budget without advancing, and the next push continues without re-pushing", async () => { + const h = await harness(); + const expected = await expectedCounts(h.db); + const total = Object.values(expected).reduce((a, b) => a + b, 0); + + // One request per second on the fake clock, 4.5 s of budget: a handful of requests. + const cut = await h.push({ budgetMs: 4_500, requestsPerSecond: 1 }); + expect(cut.state).toBe("incomplete"); + const firstPushed = cut.state === "incomplete" ? cut.pushed : -1; + expect(firstPushed).toBeGreaterThan(0); + expect(firstPushed).toBeLessThan(total); + const row = await mirrorRow(h); + expect(row.last_synced_at).toBeNull(); + expect(row.last_status).toBe("partial"); + expect(row.last_error).toMatchObject({ code: "budget_exhausted" }); + + const createsBefore = count(h, "POST", /^\/pages$/); + const rest = await h.push(); + expect(rest).toEqual({ state: "ok", pushed: total - firstPushed, archived: 0 }); + expect(count(h, "POST", /^\/pages$/) - createsBefore).toBe(total - firstPushed); + for (const entity of MIRROR_ENTITY) expect(livePages(h, entity), entity).toHaveLength(expected[entity]); + }); + + it("fails one entity whose database is gone and pushes the others", async () => { + const h = await harness(); + const expected = await expectedCounts(h.db); + h.fake.databases.delete(h.mapping.resources!); + + const outcome = await h.push(); + expect(outcome).toMatchObject({ state: "partial", failed: 0, error: { code: "database_not_found", entities: ["resources"] } }); + for (const entity of MIRROR_ENTITY.filter((entity) => entity !== "resources")) { + expect(livePages(h, entity), entity).toHaveLength(expected[entity]); + } + expect(createdEntities(h)).not.toContain(undefined); + const row = await mirrorRow(h); + expect(row.last_synced_at).toBeNull(); + expect(row.last_status).toBe("partial"); + + // Create databases recreates it; the next push fills it. + const ensured = await ensureMirrorDatabases(h.mirrorId, { db: h.db, client: { requestsPerSecond: 0 } }); + if (!ensured.ok) throw new Error(ensured.code); + h.mapping = ensured.mapping; + expect(await h.push()).toMatchObject({ state: "ok", pushed: expected.resources }); + expect(livePages(h, "resources")).toHaveLength(expected.resources); + }); + + it("after the Tools database is recreated, re-pushes every page that links to a tool", async () => { + const h = await harness(); + expect(await h.push()).toMatchObject({ state: "ok" }); + const oldToolsDatabase = h.mapping.tools!; + const oldToolPages = new Set(h.fake.pagesIn(oldToolsDatabase).map((page) => page.id)); + h.fake.databases.delete(oldToolsDatabase); + + expect(await h.push()).toEqual({ state: "ok", pushed: 0, archived: 0 }); + const ensured = await ensureMirrorDatabases(h.mirrorId, { db: h.db, client: { requestsPerSecond: 0 } }); + if (!ensured.ok) throw new Error(ensured.code); + expect(ensured.created).toEqual(["tools"]); + h.mapping = ensured.mapping; + + expect(await h.push()).toMatchObject({ state: "ok" }); + const newToolPages = new Set(livePages(h, "tools").map((page) => page.id)); + expect(newToolPages.size).toBeGreaterThan(0); + + // Every Tool / Tools relation now points into the new database, none into the old. + const linked = (entity: MirrorEntity, property: string) => + livePages(h, entity).flatMap((page) => + ((page.properties[property] as { relation?: { id: string }[] } | undefined)?.relation ?? []).map((r) => r.id) + ); + for (const [entity, property] of [ + ["units", "Tool"], + ["resources", "Tool"], + ["maintenance", "Tool"], + ["projects", "Tools"], + ] as const) { + const ids = linked(entity, property); + expect(ids.length, entity).toBeGreaterThan(0); + for (const id of ids) { + expect(oldToolPages.has(id), `${entity} still links to an old tool page`).toBe(false); + expect(newToolPages.has(id), entity).toBe(true); + } + } + const row = await mirrorRow(h); + expect(row.last_status).toBe("ok"); + }); + + it("a mapping reset during the push stops it without advancing or recording pages into the old database", async () => { + const h = await harness(); + let calls = 0; + h.sleep.mockImplementation(async (ms: number) => { + h.clock.t += ms; + calls += 1; + // A few requests in: Create databases recreates the categories database. + if (calls === 3) await resetMirrorEntities(h.mirrorId, ["categories"], { db: h.db }); + }); + + const outcome = await h.push({ requestsPerSecond: 3 }); + + expect(outcome.state).toBe("incomplete"); + const row = await mirrorRow(h); + expect(row.last_synced_at).toBeNull(); + expect(row.running_since).toBeNull(); + const [recorded] = await rawRows<{ n: number }>( + h.db, + sql`select count(*)::int as n from mirror_pages where mirror_id = ${h.mirrorId} and entity = 'categories'` + ); + expect(Number(recorded.n)).toBe(0); + }); + + it("recreates a page deleted by hand", async () => { + const h = await harness(); + await h.push(); + const [category] = await h.db.select({ id: categories.id }).from(categories).limit(1); + const [page] = await h.db.select({ id: mirrorPages.notionPageId }).from(mirrorPages).where(eq(mirrorPages.entityId, category.id)); + h.fake.pages.delete(page.id); + await h.db.update(categories).set({ name: "Touched" }).where(eq(categories.id, category.id)); + + expect(await h.push()).toEqual({ state: "ok", pushed: 1, archived: 0 }); + const [again] = await h.db.select({ id: mirrorPages.notionPageId }).from(mirrorPages).where(eq(mirrorPages.entityId, category.id)); + expect(again.id).not.toBe(page.id); + expect(h.fake.pages.has(again.id)).toBe(true); + }); + + it("archives an orphaned unit page and forgets it", async () => { + const h = await harness(); + await h.push(); + const [unit] = await h.db.select({ id: units.id }).from(units).limit(1); + const [page] = await h.db.select({ id: mirrorPages.notionPageId }).from(mirrorPages).where(eq(mirrorPages.entityId, unit.id)); + await h.db.delete(units).where(eq(units.id, unit.id)); + + const outcome = await h.push(); + expect(outcome).toMatchObject({ state: "ok", archived: 1 }); + expect(h.fake.pages.get(page.id)!.archived).toBe(true); + expect(await h.db.select().from(mirrorPages).where(eq(mirrorPages.entityId, unit.id))).toEqual([]); + }); + + it("releases the guard and records the failure when the database throws", async () => { + const h = await harness(); + await h.db.execute(sql`alter table maintenance_logs rename to maintenance_logs_gone`); + await expect(h.push()).rejects.toThrow(); + const row = await mirrorRow(h); + expect(row.running_since).toBeNull(); + expect(row.last_status).toBe("failed"); + expect(row.last_error).toMatchObject({ code: "unknown" }); + }); + + it("never lets the token or an email reach the console, last_error or a thrown message", async () => { + const lines: string[] = []; + for (const method of ["log", "info", "warn", "error", "debug"] as const) { + vi.spyOn(console, method).mockImplementation((...args: unknown[]) => { + lines.push(args.map(String).join(" ")); + }); + } + const h = await harness(); + const [log] = await h.db.select({ email: maintenanceLogs.reportedByEmail }).from(maintenanceLogs).where(sql`${maintenanceLogs.reportedByEmail} is not null`); + + // A Notion error whose message quotes the token and an email. + h.fake.failNext({ method: "POST", path: /^\/pages$/ }, { status: 400, message: `bad ${TOKEN} for ${log.email}` }, 2); + await h.push(); + h.fake.failNext({}, { status: 401, message: `revoked ${TOKEN}` }, 1); + await h.push(); + const errors = JSON.stringify((await getMirror(h.mirrorId, { db: h.db }))!.lastError); + + await h.db.update(notionMirrors).set({ pausedAt: null }).where(eq(notionMirrors.id, h.mirrorId)); + await h.db.execute(sql`alter table projects rename to projects_gone`); + const thrown = await h.push().catch((error: unknown) => String((error as Error)?.message ?? error)); + + expect(lines.length).toBeGreaterThan(0); + for (const text of [...lines, errors, String(thrown)]) { + expect(text).not.toContain(TOKEN); + expect(text).not.toContain("ntn_"); + expect(text).not.toContain(SECRET); + expect(text).not.toContain(log.email!); + } + }); +}); diff --git a/v5/src/lib/mirror/push.ts b/v5/src/lib/mirror/push.ts new file mode 100644 index 0000000..dfbedc8 --- /dev/null +++ b/v5/src/lib/mirror/push.ts @@ -0,0 +1,561 @@ +import { getDb } from "../db/client.ts"; +import { MIRROR_ENTITY, type MirrorEntity } from "../db/schema/vocabulary.ts"; +import type { Db } from "../db/types.ts"; +import { deleteMirrorPage, getMirrorPageIds, listOrphanedMirrorPages, upsertMirrorPage } from "../data/mirror-pages.ts"; +import { claimMirrorRun, finishMirrorRun, releaseMirrorRun, type ClaimedMirror } from "../data/mirrors.ts"; +import { mirrorClientFor } from "./credentials.ts"; +import { validateDatabaseSchema } from "./database-schemas.ts"; +import { MIRROR_PUSH_BUDGET_MS } from "./limits.ts"; +import { NotionMirrorError, scrubSecrets, type NotionClient, type NotionClientOptions } from "./notion-client.ts"; +import { buildMirrorProperties, relationTargets, type MirrorRelations } from "./properties.ts"; +import { listSourceRows, SOURCE_PAGE_SIZE, type AnySourceRow, type SourceCursor } from "./source.ts"; +import type { MirrorErrorCode, MirrorLastError, MirrorMapping } from "./types.ts"; + +/** + * One push of one mirror (spec §3.8 "Push", §5.8, §8 "External calls"). + * + * 1. **Claim** (`claimMirrorRun`): the overlap guard. A paused, disconnected + * or already-running mirror is skipped before a single request is made. + * 2. **Open a client** from the stored ciphertext. An unset `AUTH_SECRET` is a + * failed push that stays active (setting the secret fixes it); a token the + * current key cannot read pauses the mirror, since only reconnecting can. + * 3. **For each mapped entity in dependency order**, page through the rows + * `source.ts` selects. Before the first write to an entity its database is + * read once and checked against the schema, so a deleted database is + * `database_not_found` for that entity alone and the push moves on. Each + * row updates its page (or, if Notion says the page is gone, creates a new + * one), or creates one and records it at once, so a push cut short never + * creates a page twice. An archived tool or unpublished project archives + * its page. A row whose relation target has no page yet is pushed without + * it and recorded as not mirrored, so the next push tries again. + * 4. **Archive orphans** — pages whose row was deleted — and forget them. + * 5. **Finish** (`finishMirrorRun`). Only a clean, complete push advances + * `last_synced_at`, to the claim's watermark; a partial or failed one leaves + * it where it was so the rows that failed are selected again. + * + * **The mapping may change under a push** (Create databases, Save mapping bump + * `mapping_generation`). The claim carries the generation it started under; + * a page is recorded, and `last_synced_at` advanced, only while the mirror is + * still at it. A push that finds it moved stops as `superseded`, frees the + * guard without recording a result, and answers `incomplete`, so the workflow + * runs another round from the new mapping. + * + * **Stops.** A 401 stops everything and pauses the mirror (§5.8). The 45 s + * budget, or a 429 whose wait would pass it, stops the loop with what is done + * recorded; three Notion 5xx responses in a row stop it as + * `notion_unavailable`, since a Notion that is down would fail every row. + * + * **Nothing secret or personal is logged or stored as an error.** One + * `console.info` line per push carries the mirror id and counts. The + * `last_error.detail` is generic English built here — entity names, counts and + * Notion's status and error code — run through `scrubSecrets`; it never holds + * Notion's message, row content, a name or an email. + * + * Throws only on a database error, and releases `running_since` in a `finally` + * even then. + * + * Relative imports with `.ts` extensions, no `server-only`: this is step code. + */ + +export interface PushMirrorOptions { + db?: Db; + budgetMs?: number; + client?: Partial>; +} + +export type MirrorPushOutcome = + | { + state: "skipped"; + reason: "not_found" | "paused" | "not_connected" | "owner_not_allowed" | "running" | "not_mapped"; + } + | { state: "ok"; pushed: number; archived: number } + /** + * More to push: the budget ran out with no failures, or the mapping changed + * under the push (Create databases, Save mapping) and the next round must + * run from the new one. + */ + | { state: "incomplete"; pushed: number; archived: number } + | { state: "partial"; pushed: number; archived: number; failed: number; error: MirrorLastError } + | { state: "failed"; error: MirrorLastError; paused: boolean }; + +/** Consecutive Notion 5xx / network failures after which the push stops. */ +const MAX_CONSECUTIVE_UNAVAILABLE = 3; +/** How many orphaned pages one pass archives per entity. */ +const ORPHAN_BATCH = 50; +const MAX_DETAIL = 300; + +/** Why the loop stopped early. `superseded`: the mapping changed under the push. */ +type Stop = "budget" | "unauthorized" | "unavailable" | "superseded"; + +/** What a push has done so far. */ +interface RunState { + pushed: number; + archived: number; + failedRows: number; + /** Entities with a failed row, a missing relation, or a failed database. */ + failedEntities: Set; + missingDatabases: Set; + schemaMismatches: Map; + /** The last Notion status and code a row failed with — for the detail line. */ + lastRowError: { status: number | null; code: string | null } | null; + consecutiveUnavailable: number; + stop: Stop | null; +} + +/** Thrown inside the loop to stop it; never escapes {@link pushMirror}. */ +class StopPush extends Error { + stop: Stop; + constructor(stop: Stop) { + super(`mirror push stopped: ${stop}`); + this.stop = stop; + } +} + +export async function pushMirror(mirrorId: string, options: PushMirrorOptions = {}): Promise { + const db = options.db ?? (await getDb()); + const claim = await claimMirrorRun(mirrorId, { db }); + if ("skipped" in claim) { + const outcome: MirrorPushOutcome = { state: "skipped", reason: claim.skipped }; + logOutcome(mirrorId, outcome); + return outcome; + } + + let released = false; + try { + const outcome = await run(claim, options, db); + released = true; + logOutcome(mirrorId, outcome); + return outcome; + } finally { + if (!released) { + // A database error (or a bug) escaped: record it and free the guard, so + // the next trigger does not wait fifteen minutes for a push that died. + // If the database is what failed, this fails too; the original error is + // the one that propagates, and the 15-minute staleness rule frees it. + try { + await finishMirrorRun( + mirrorId, + { + status: "failed", + error: lastError("unknown", [], 0, "A database error interrupted the push."), + advanceTo: null, + pause: false, + generation: claim.generation, + }, + { db } + ); + } catch { + // Nothing more to do; see above. + } + } + } +} + +/** Everything after the claim. Always finishes the run itself, except when it throws. */ +async function run(claim: ClaimedMirror, options: PushMirrorOptions, db: Db): Promise { + const mirrorId = claim.id; + const mapping = claim.mapping; + const entities = MIRROR_ENTITY.filter((entity) => mapping[entity]); + if (entities.length === 0) { + await releaseMirrorRun(mirrorId, { db }); + return { state: "skipped", reason: "not_mapped" }; + } + + const now = options.client?.now ?? Date.now; + const budgetMs = Math.max(0, options.budgetMs ?? MIRROR_PUSH_BUDGET_MS); + const deadline = now() + budgetMs; + const opened = mirrorClientFor(claim.tokenCiphertext, { ...options.client, deadline }); + if (!opened.ok) { + // `not_connected` cannot happen after a claim (it requires a token); it is + // reported as unreadable rather than inventing a state for it. + const code: MirrorErrorCode = opened.code === "key_unavailable" ? "key_unavailable" : "token_unreadable"; + const pause = code === "token_unreadable"; + const error = lastError( + code, + [], + 0, + code === "key_unavailable" + ? "AUTH_SECRET is not set, so the stored token cannot be decrypted." + : "The stored token cannot be decrypted with the current key. Connect again." + ); + await finishMirrorRun(mirrorId, { status: "failed", error, advanceTo: null, pause, generation: claim.generation }, { db }); + return { state: "failed", error, paused: pause }; + } + const client = opened.client; + + const state: RunState = { + pushed: 0, + archived: 0, + failedRows: 0, + failedEntities: new Set(), + missingDatabases: new Set(), + schemaMismatches: new Map(), + lastRowError: null, + consecutiveUnavailable: 0, + stop: null, + }; + + try { + for (const entity of entities) { + await pushEntity(entity, { claim, client, db, state, mapping }); + } + for (const entity of entities) { + if (state.missingDatabases.has(entity) || state.schemaMismatches.has(entity)) continue; + await archiveOrphans(entity, { claim, client, db, state, mapping }); + } + } catch (error) { + if (!(error instanceof StopPush)) throw error; + state.stop = error.stop; + } + + const generation = claim.generation; + if (state.stop === "unauthorized") { + const error = lastError("unauthorized", [], state.failedRows, "Notion refused the token (401). Connect again with a new token."); + await finishMirrorRun(mirrorId, { status: "failed", error, advanceTo: null, pause: true, generation }, { db }); + return { state: "failed", error, paused: true }; + } + if (state.stop === "superseded") { + // The mapping changed under this push. What it did is not a result for + // the new mapping — free the guard, keep the result the mirror last + // earned, and let the workflow run the next round from the new mapping. + await releaseMirrorRun(mirrorId, { db }); + return { state: "incomplete", pushed: state.pushed, archived: state.archived }; + } + + const failures = state.failedRows > 0 || state.failedEntities.size > 0 || state.stop === "unavailable"; + if (!failures && state.stop === null) { + const finished = await finishMirrorRun( + mirrorId, + { status: "ok", error: null, advanceTo: claim.watermark, pause: false, generation }, + { db } + ); + // Clean, but under a mapping that has since changed: `last_synced_at` + // stayed where the change put it, and another round pushes the rest. + if (!finished.current) return { state: "incomplete", pushed: state.pushed, archived: state.archived }; + return { state: "ok", pushed: state.pushed, archived: state.archived }; + } + if (!failures) { + const seconds = Math.round(budgetMs / 1000); + const error = lastError( + "budget_exhausted", + [], + 0, + `The ${seconds}-second budget for one push ran out after ${state.pushed + state.archived} page writes; the rest waits for the next push.` + ); + await finishMirrorRun(mirrorId, { status: "partial", error, advanceTo: null, pause: false, generation }, { db }); + return { state: "incomplete", pushed: state.pushed, archived: state.archived }; + } + + const error = failureError(state); + const anything = state.pushed + state.archived > 0; + await finishMirrorRun( + mirrorId, + { status: anything ? "partial" : "failed", error, advanceTo: null, pause: false, generation }, + { db } + ); + if (anything) { + return { state: "partial", pushed: state.pushed, archived: state.archived, failed: state.failedRows, error }; + } + return { state: "failed", error, paused: false }; +} + +interface EntityContext { + claim: ClaimedMirror; + client: NotionClient; + db: Db; + state: RunState; + mapping: MirrorMapping; +} + +/** Push every changed row of one entity. */ +async function pushEntity(entity: MirrorEntity, ctx: EntityContext): Promise { + const { claim, client, db, state, mapping } = ctx; + const databaseId = mapping[entity]!; + let checked = false; + let after: SourceCursor | null = null; + + for (;;) { + const rows: AnySourceRow[] = await listSourceRows(entity, { + mirrorId: claim.id, + since: claim.since, + after, + limit: SOURCE_PAGE_SIZE, + db, + }); + if (rows.length === 0) return; + const last = rows[rows.length - 1]; + after = { revision: last.revision, id: last.id }; + + if (!checked) { + if (!(await databaseUsable(entity, databaseId, ctx))) return; + checked = true; + } + + const relations = await relationsFor(entity, rows, ctx); + for (const row of rows) { + if (client.remainingMs() <= 0) throw new StopPush("budget"); + const result = await pushRow(entity, databaseId, row, relations, ctx); + if (result === "database_gone") { + state.missingDatabases.add(entity); + state.failedEntities.add(entity); + return; + } + } + if (rows.length < SOURCE_PAGE_SIZE) return; + } +} + +/** + * Read the entity's database once before writing to it. A missing, archived + * or trashed database fails the entity as `database_not_found`; one without + * the expected properties as `schema_mismatch`. Either way the push goes on + * to the next entity. + */ +async function databaseUsable(entity: MirrorEntity, databaseId: string, ctx: EntityContext): Promise { + const { client, state, mapping } = ctx; + try { + const database = await client.getDatabase(databaseId); + noteSuccess(state); + if (database.archived === true || database.in_trash === true) { + state.missingDatabases.add(entity); + state.failedEntities.add(entity); + return false; + } + const problem = validateDatabaseSchema(entity, database, mapping); + if (problem) { + state.schemaMismatches.set(entity, [...(problem.missing ?? []), ...(problem.wrongType ?? [])]); + state.failedEntities.add(entity); + return false; + } + return true; + } catch (error) { + const notion = asNotionError(error); + if (notion.code === "database_not_found" || notion.code === "not_found") { + state.missingDatabases.add(entity); + state.failedEntities.add(entity); + return false; + } + handleStopping(notion); + // Anything else (a 5xx, a 403): the entity cannot be pushed this time. + noteRowFailure(entity, notion, state, 0); + return false; + } +} + +/** `mirror_pages` lookups for every relation target `rows` name. */ +async function relationsFor(entity: MirrorEntity, rows: AnySourceRow[], ctx: EntityContext): Promise { + const pages = new Map>(); + for (const [target, ids] of relationTargets(entity, rows as never[])) { + if (!ctx.mapping[target]) continue; + pages.set(target, await getMirrorPageIds(ctx.claim.id, target, [...ids], { db: ctx.db })); + } + return { + mapped: (target) => Boolean(ctx.mapping[target]), + pageId: (target, id) => pages.get(target)?.get(id) ?? null, + }; +} + +/** + * Push one row. Returns `database_gone` when Notion says the entity's database + * no longer exists, so the caller abandons the entity; otherwise records + * the outcome in `state` itself. + */ +async function pushRow( + entity: MirrorEntity, + databaseId: string, + row: AnySourceRow, + relations: MirrorRelations, + ctx: EntityContext +): Promise<"done" | "database_gone"> { + const { client, db, state, claim } = ctx; + try { + if (row.archive) { + if (!row.pageId) return "done"; + try { + await client.updatePage(row.pageId, { archived: true }); + } catch (error) { + if (asNotionError(error).code !== "page_not_found") throw error; + // Deleted by hand already: nothing to archive, nothing to remember. + await deleteMirrorPage(claim.id, entity, row.id, { db }); + noteSuccess(state); + return "done"; + } + await recordPage(ctx, { entity, entityId: row.id, notionPageId: row.pageId, sourceUpdatedAt: row.revision }); + state.archived += 1; + noteSuccess(state); + return "done"; + } + + const built = buildMirrorProperties(entity, row as never, relations); + let pageId = row.pageId; + if (pageId) { + try { + await client.updatePage(pageId, { properties: built.properties, archived: false }); + } catch (error) { + if (asNotionError(error).code !== "page_not_found") throw error; + // The page was deleted by hand: forget it and create a new one. + await deleteMirrorPage(claim.id, entity, row.id, { db }); + pageId = null; + } + } + if (!pageId) { + const created = await client.createPage({ parent: { database_id: databaseId }, properties: built.properties }); + pageId = created.id; + } + await recordPage(ctx, { + entity, + entityId: row.id, + notionPageId: pageId, + // A row pushed without a relation it should have is not mirrored yet. + sourceUpdatedAt: built.missingRelation ? null : row.revision, + }); + state.pushed += 1; + noteSuccess(state); + if (built.missingRelation) { + state.failedRows += 1; + state.failedEntities.add(entity); + } + return "done"; + } catch (error) { + if (!(error instanceof NotionMirrorError)) throw error; // a database error: let it propagate + if (error.code === "database_not_found") return "database_gone"; + handleStopping(error); + noteRowFailure(entity, error, state, 1); + return "done"; + } +} + +/** + * Record a row's page under the claim's generation. When the mapping changed + * during the push the row is not written — its page is in a database the + * mirror no longer maps — and the push stops as `superseded`. + */ +async function recordPage( + ctx: EntityContext, + page: { entity: MirrorEntity; entityId: string; notionPageId: string; sourceUpdatedAt: string | null } +): Promise { + const recorded = await upsertMirrorPage( + { mirrorId: ctx.claim.id, ...page, generation: ctx.claim.generation }, + { db: ctx.db } + ); + if (!recorded) throw new StopPush("superseded"); +} + +/** Archive and forget the pages of rows that no longer exist. */ +async function archiveOrphans(entity: MirrorEntity, ctx: EntityContext): Promise { + const { client, db, state, claim } = ctx; + for (;;) { + const orphans = await listOrphanedMirrorPages(claim.id, entity, { limit: ORPHAN_BATCH, db }); + if (orphans.length === 0) return; + let failedAny = false; + for (const orphan of orphans) { + if (client.remainingMs() <= 0) throw new StopPush("budget"); + try { + await client.updatePage(orphan.notionPageId, { archived: true }); + state.archived += 1; + noteSuccess(state); + } catch (error) { + const notion = asNotionError(error); + if (notion.code !== "page_not_found") { + handleStopping(notion); + noteRowFailure(entity, notion, state, 1); + failedAny = true; + continue; + } + } + await deleteMirrorPage(claim.id, entity, orphan.entityId, { db }); + } + // A failed archive stays in the list; stop rather than loop on it. + if (failedAny || orphans.length < ORPHAN_BATCH) return; + } +} + +// ── State helpers ─────────────────────────────────────────────────── + +function asNotionError(error: unknown): NotionMirrorError { + if (error instanceof NotionMirrorError) return error; + throw error; +} + +/** A 401 stops everything; the budget or a 429 past it stops the loop. */ +function handleStopping(error: NotionMirrorError): void { + if (error.code === "unauthorized") throw new StopPush("unauthorized"); + if (error.code === "deadline" || error.code === "rate_limited") throw new StopPush("budget"); +} + +function noteSuccess(state: RunState): void { + state.consecutiveUnavailable = 0; +} + +function noteRowFailure(entity: MirrorEntity, error: NotionMirrorError, state: RunState, rows: number): void { + state.failedRows += rows; + state.failedEntities.add(entity); + state.lastRowError = { status: error.status, code: error.notionCode }; + if (error.code === "unavailable") { + state.consecutiveUnavailable += 1; + if (state.consecutiveUnavailable >= MAX_CONSECUTIVE_UNAVAILABLE) throw new StopPush("unavailable"); + } else { + state.consecutiveUnavailable = 0; + } +} + +/** The single error a run with failures records, most actionable first. */ +function failureError(state: RunState): MirrorLastError { + const entities = MIRROR_ENTITY.filter((entity) => state.failedEntities.has(entity)); + if (state.missingDatabases.size) { + const missing = MIRROR_ENTITY.filter((entity) => state.missingDatabases.has(entity)); + return lastError( + "database_not_found", + entities, + state.failedRows, + `Notion could not find the ${list(missing)} database${missing.length > 1 ? "s" : ""}. Create databases recreates only the missing ones.${rowsNote(state)}` + ); + } + if (state.schemaMismatches.size) { + const parts = [...state.schemaMismatches].map(([entity, names]) => `${entity} (${names.join(", ")})`); + return lastError( + "schema_mismatch", + entities, + state.failedRows, + `These databases lack properties the push writes, or have them with the wrong type: ${parts.join("; ")}.${rowsNote(state)}` + ); + } + if (state.stop === "unavailable") { + return lastError( + "notion_unavailable", + entities, + state.failedRows, + `Notion did not respond (${MAX_CONSECUTIVE_UNAVAILABLE} server errors in a row); the push stopped.${rowsNote(state)}` + ); + } + return lastError("rows_failed", entities, state.failedRows, rowsNote(state).trim() || "Some rows could not be pushed."); +} + +function rowsNote(state: RunState): string { + if (state.failedRows === 0) return ""; + const entities = MIRROR_ENTITY.filter((entity) => state.failedEntities.has(entity)); + const last = state.lastRowError; + const notion = last ? ` Last Notion response: ${last.status ?? "no status"}${last.code ? ` ${last.code}` : ""}.` : ""; + return ` ${state.failedRows} row${state.failedRows === 1 ? "" : "s"} could not be mirrored (${list(entities)}) and will be tried again.${notion}`; +} + +function list(entities: readonly string[]): string { + return entities.join(", "); +} + +function lastError(code: MirrorErrorCode, entities: MirrorEntity[], failed: number, detail: string): MirrorLastError { + const clean = scrubSecrets(detail).replace(/\s+/g, " ").trim().slice(0, MAX_DETAIL); + return { code, entities, failed, detail: clean || null }; +} + +function logOutcome(mirrorId: string, outcome: MirrorPushOutcome): void { + const counts = + outcome.state === "skipped" + ? `reason=${outcome.reason}` + : outcome.state === "failed" + ? `code=${outcome.error.code} paused=${outcome.paused}` + : outcome.state === "partial" + ? `pushed=${outcome.pushed} archived=${outcome.archived} failed=${outcome.failed} code=${outcome.error.code}` + : `pushed=${outcome.pushed} archived=${outcome.archived}`; + console.info(`[mirror] push ${mirrorId}: ${outcome.state} ${counts}`); +} diff --git a/v5/src/lib/mirror/source.test.ts b/v5/src/lib/mirror/source.test.ts new file mode 100644 index 0000000..bcd9f6e --- /dev/null +++ b/v5/src/lib/mirror/source.test.ts @@ -0,0 +1,229 @@ +// @vitest-environment node +import { eq, sql } from "drizzle-orm"; +import { createPgliteDb } from "../db/pglite"; +import { rawRows } from "../db/raw"; +import { + attachments, + categories, + maintenanceLogs, + projectTools, + projects, + tools, + units, + user, +} from "../db/schema/index"; +import type { MirrorEntity } from "../db/schema/vocabulary"; +import type { Db } from "../db/types"; +import { upsertMirrorPage } from "../data/mirror-pages"; +import { saveMirrorConnection } from "../data/mirrors"; +import { listSourceRows } from "./source"; + +/** + * What a push selects (spec §3.8 "What is mirrored", 2026-09-23 amendment), + * against PGlite. Each test gets its own database so row counts are exact. + */ + +const MICROSECOND_STAMP = "2026-01-01 10:00:00.123456+00"; + +async function setup(): Promise<{ db: Db; mirrorId: string }> { + const db = await createPgliteDb(); + const owner = `u-${crypto.randomUUID()}`; + await db.insert(user).values({ id: owner, name: "Owner", email: `${owner}@cornell.edu`, role: "admin" }); + const { mirror } = await saveMirrorConnection( + { ownerUserId: owner, tokenCiphertext: new Uint8Array([1, 2, 3]), parentPageId: crypto.randomUUID(), parentPageTitle: null }, + { db } + ); + return { db, mirrorId: mirror.id }; +} + +async function insertTool(db: Db, values: Partial = {}): Promise { + const [row] = await db + .insert(tools) + .values({ slug: `t-${crypto.randomUUID()}`, name: "Form 4", ...values }) + .returning({ id: tools.id }); + return row.id; +} + +async function all(db: Db, mirrorId: string, entity: E, since: string | null = null) { + return listSourceRows(entity, { db, mirrorId, since, limit: 500 }); +} + +describe("mirror source rows", () => { + it("includes drafts, and an archived tool only when it has a page to archive", async () => { + const { db, mirrorId } = await setup(); + const published = await insertTool(db, { published: true }); + const draft = await insertTool(db, { published: false }); + const archivedUnmirrored = await insertTool(db, { published: true, archivedAt: new Date() }); + const archivedMirrored = await insertTool(db, { published: true, archivedAt: new Date() }); + await upsertMirrorPage( + { mirrorId, entity: "tools", entityId: archivedMirrored, notionPageId: "page-archived", sourceUpdatedAt: null }, + { db } + ); + + const rows = await all(db, mirrorId, "tools"); + const byId = new Map(rows.map((row) => [row.id, row])); + expect(byId.get(published)).toMatchObject({ published: true, archive: false, pageId: null }); + expect(byId.get(draft)).toMatchObject({ published: false, archive: false }); + expect(byId.has(archivedUnmirrored)).toBe(false); + expect(byId.get(archivedMirrored)).toMatchObject({ archive: true, archived: true, pageId: "page-archived" }); + }); + + it("selects only public attachments with a URL, in position order — never a private maintenance photo", async () => { + const { db, mirrorId } = await setup(); + const tool = await insertTool(db); + const [log] = await db + .insert(maintenanceLogs) + .values({ title: "Leak", toolId: tool, reportedByName: "Ada", reportedByEmail: "ada@cornell.edu" }) + .returning({ id: maintenanceLogs.id }); + await db.insert(attachments).values([ + { ownerType: "tool", ownerId: tool, position: 1, blobPathname: "tools/second.jpg", access: "public", publicUrl: "https://blob.example/second.jpg" }, + { ownerType: "tool", ownerId: tool, position: 0, blobPathname: "tools/first.jpg", access: "public", publicUrl: "https://blob.example/first.jpg", originalFilename: "first.jpg" }, + { ownerType: "tool", ownerId: tool, position: 2, blobPathname: "tools/private.jpg", access: "private", publicUrl: "https://blob.example/tool-private.jpg" }, + { ownerType: "tool", ownerId: tool, position: 3, blobPathname: "tools/no-url.jpg", access: "public", publicUrl: null }, + { ownerType: "maintenance_log", ownerId: log.id, position: 0, blobPathname: "maintenance/PRIVATE-PHOTO.jpg", access: "private", publicUrl: null }, + // Even a maintenance photo that somehow had a public URL is never read. + { ownerType: "maintenance_log", ownerId: log.id, position: 1, blobPathname: "maintenance/odd.jpg", access: "public", publicUrl: "https://blob.example/MAINTENANCE-PHOTO.jpg" }, + ]); + + const [toolRow] = await all(db, mirrorId, "tools"); + expect(toolRow.images).toEqual([ + { url: "https://blob.example/first.jpg", name: "first.jpg" }, + { url: "https://blob.example/second.jpg", name: null }, + ]); + + const everything = JSON.stringify( + await Promise.all( + (["categories", "locations", "tools", "units", "resources", "maintenance", "projects"] as const).map((entity) => + all(db, mirrorId, entity) + ) + ) + ); + expect(everything).not.toContain("PRIVATE-PHOTO"); + expect(everything).not.toContain("MAINTENANCE-PHOTO"); + expect(everything).not.toContain("tool-private"); + }); + + it("carries the maintenance reporter's and assignee's names and emails", async () => { + const { db, mirrorId } = await setup(); + await db.insert(user).values({ id: "assignee-1", name: "Niti", email: "niti@cornell.edu" }); + const tool = await insertTool(db); + await db.insert(maintenanceLogs).values({ + title: "Leak", + toolId: tool, + reportedByName: "Ada", + reportedByEmail: "ada@cornell.edu", + assignedToUserId: "assignee-1", + assignedToName: "Niti", + dateReported: "2026-02-01", + }); + const [row] = await all(db, mirrorId, "maintenance"); + expect(row).toMatchObject({ + reportedByName: "Ada", + reportedByEmail: "ada@cornell.edu", + assignedToName: "Niti", + assigneeEmail: "niti@cornell.edu", + toolId: tool, + dateReported: "2026-02-01", + }); + }); + + it("excludes unpublished projects unless already mirrored, carries the author's email, and leaves archived tools out", async () => { + const { db, mirrorId } = await setup(); + await db.insert(user).values({ id: "author-1", name: "Luis", email: "luis@cornell.edu" }); + const liveTool = await insertTool(db); + const archivedTool = await insertTool(db, { archivedAt: new Date() }); + const [live, waiting, withdrawn] = await db + .insert(projects) + .values([ + { slug: "live", title: "Live", published: true, authorUserId: "author-1", authorName: "Luis" }, + { slug: "waiting", title: "Waiting", published: false }, + { slug: "withdrawn", title: "Withdrawn", published: false }, + ]) + .returning({ id: projects.id }); + await db.insert(projectTools).values([ + { projectId: live.id, toolId: liveTool }, + { projectId: live.id, toolId: archivedTool }, + ]); + await upsertMirrorPage({ mirrorId, entity: "projects", entityId: withdrawn.id, notionPageId: "page-w", sourceUpdatedAt: null }, { db }); + + const rows = await all(db, mirrorId, "projects"); + const ids = rows.map((row) => row.id); + expect(ids).toContain(live.id); + expect(ids).not.toContain(waiting.id); + expect(rows.find((row) => row.id === withdrawn.id)).toMatchObject({ archive: true, pageId: "page-w" }); + expect(rows.find((row) => row.id === live.id)).toMatchObject({ + archive: false, + authorName: "Luis", + authorEmail: "luis@cornell.edu", + toolIds: [liveTool], + }); + }); + + it("drops the relation to an archived tool from units", async () => { + const { db, mirrorId } = await setup(); + const archivedTool = await insertTool(db, { archivedAt: new Date() }); + const liveTool = await insertTool(db); + await db.insert(units).values([ + { toolId: archivedTool, unitLabel: "Old #1" }, + { toolId: liveTool, unitLabel: "New #1" }, + ]); + const rows = await all(db, mirrorId, "units"); + expect(Object.fromEntries(rows.map((row) => [row.unitLabel, row.toolId]))).toEqual({ "Old #1": null, "New #1": liveTool }); + }); + + it("keeps the revision's microseconds", async () => { + const { db, mirrorId } = await setup(); + const id = await insertTool(db); + await db.execute(sql`alter table tools disable trigger tools_set_updated_at`); + await db.execute(sql`update tools set updated_at = ${sql.raw(`timestamptz '${MICROSECOND_STAMP}'`)} where id = ${id}`); + await db.execute(sql`alter table tools enable trigger tools_set_updated_at`); + + const [row] = await all(db, mirrorId, "tools"); + expect(row.revision).toMatch(/\.123456/); + const [check] = await rawRows<{ same: boolean }>( + db, + sql`select (${row.revision}::timestamptz = ${sql.raw(`timestamptz '${MICROSECOND_STAMP}'`)}) as same` + ); + expect(check.same).toBe(true); + + // Written back verbatim, the row is current: nothing is selected. + await upsertMirrorPage({ mirrorId, entity: "tools", entityId: id, notionPageId: "p", sourceUpdatedAt: row.revision }, { db }); + expect(await all(db, mirrorId, "tools")).toEqual([]); + }); + + it("skips rows already mirrored at their revision, and selects them again when they change", async () => { + const { db, mirrorId } = await setup(); + const [a, b] = await db + .insert(categories) + .values([{ name: "A" }, { name: "B" }]) + .returning({ id: categories.id }); + const first = await all(db, mirrorId, "categories"); + expect(first.map((row) => row.name).sort()).toEqual(["A", "B"]); + + const rowA = first.find((row) => row.id === a.id)!; + await upsertMirrorPage({ mirrorId, entity: "categories", entityId: a.id, notionPageId: "page-a", sourceUpdatedAt: rowA.revision }, { db }); + // A page recorded without a revision (a deferred row) is selected again. + await upsertMirrorPage({ mirrorId, entity: "categories", entityId: b.id, notionPageId: "page-b", sourceUpdatedAt: null }, { db }); + const second = await all(db, mirrorId, "categories"); + expect(second.map((row) => [row.name, row.pageId])).toEqual([["B", "page-b"]]); + + await db.update(categories).set({ group: "Fabrication" }).where(eq(categories.id, a.id)); + const third = await all(db, mirrorId, "categories"); + expect(third.map((row) => row.name).sort()).toEqual(["A", "B"]); + expect(third.find((row) => row.id === a.id)).toMatchObject({ group: "Fabrication", pageId: "page-a" }); + }); + + it("honours since, and pages through with a keyset cursor", async () => { + const { db, mirrorId } = await setup(); + await db.insert(categories).values([{ name: "A" }, { name: "B" }, { name: "C" }]); + const [{ later }] = await rawRows<{ later: string }>(db, sql`select (now() + interval '1 minute')::text as later`); + expect(await all(db, mirrorId, "categories", later)).toEqual([]); + + const page1 = await listSourceRows("categories", { db, mirrorId, since: null, limit: 2 }); + expect(page1).toHaveLength(2); + const last = page1[1]; + const page2 = await listSourceRows("categories", { db, mirrorId, since: null, limit: 2, after: { revision: last.revision, id: last.id } }); + expect(page2).toHaveLength(1); + expect(new Set([...page1, ...page2].map((row) => row.name))).toEqual(new Set(["A", "B", "C"])); + }); +}); diff --git a/v5/src/lib/mirror/source.ts b/v5/src/lib/mirror/source.ts new file mode 100644 index 0000000..ea78fb1 --- /dev/null +++ b/v5/src/lib/mirror/source.ts @@ -0,0 +1,526 @@ +import { sql, type SQL } from "drizzle-orm"; +import { getDb } from "../db/client.ts"; +import { rawRows } from "../db/raw.ts"; +import type { MirrorEntity } from "../db/schema/vocabulary.ts"; +import type { Db } from "../db/types.ts"; + +/** + * The rows a push sends, per entity (spec §3.8 "Push" step 2, "What is + * mirrored"; 2026-09-23 amendment). + * + * Each read is one SELECT over the entity's table, LEFT JOINed to this + * mirror's `mirror_pages`, returning only rows that + * + * - changed since the last complete push (`updated_at > since`, or every row + * when `since` is null), **and** + * - are not already mirrored at this revision: no page, a page recorded with + * no `source_updated_at` (a row deferred for a missing relation), or + * `updated_at > source_updated_at`. + * + * The second test is what lets a push cut short by its budget, or re-reading + * the watermark's five-minute margin, skip everything it already sent. + * + * **Timestamps never become a JavaScript `Date`** (read `data/revision.ts`). + * `revision` is `updated_at::text`, which keeps Postgres' microseconds, and + * the push writes it back verbatim as `source_updated_at` with + * `$::timestamptz`. A `Date` would lose the microseconds on Neon, every row + * would compare newer than its own page, and the mirror would re-push its + * whole inventory forever. The keyset cursor is the same text. + * + * What is selected, and what is not: + * + * - **Tools:** every tool, drafts included. An archived tool is selected only + * when it already has a page, flagged `archive` — its page is archived, and + * a tool that was never mirrored stays unmirrored. + * - **Projects:** published ones, plus an unpublished one that already has a + * page, flagged `archive`. + * - **Emails are carried** (2026-09-23 amendment): a maintenance log's + * reporter name and email and its assignee's name and account email; a + * project's author name and account email. They go to Notion and nowhere + * else — this module never logs, and the push logs counts only. + * - **Files:** only attachments with `access = 'public'` and a public URL, + * owned by the row itself (a tool, a resource or a project), in `position` + * order. Nothing reads `maintenance_log` attachments, which are private + * photos, so there is no query here that could select one. + * - **A relation to an archived tool is dropped** in SQL (`tool_id` comes back + * null), so a unit, resource or maintenance log never links to a page the + * mirror has archived; a project's tool list leaves archived tools out. + * + * Relative imports with `.ts` extensions, no `server-only`: workflow step code + * loads this module. + */ + +// ── Shapes ────────────────────────────────────────────────────────── + +/** A public file, as a files property takes it. */ +export interface MirrorFile { + url: string; + /** The uploaded file's name, when one was recorded. */ + name: string | null; +} + +export interface SourceRowBase { + id: string; + /** `updated_at::text` — opaque; written back as `source_updated_at` and used as the cursor. */ + revision: string; + /** `updated_at` as ISO-8601 UTC, for the `Updated` property. */ + updatedAt: string; + /** The Notion page that already mirrors this row, or null. */ + pageId: string | null; + /** Archive this row's page instead of updating it (an archived tool, an unpublished project). */ + archive: boolean; +} + +export interface CategorySourceRow extends SourceRowBase { + name: string; + group: string | null; +} + +export interface LocationSourceRow extends SourceRowBase { + room: string; + zone: string; + mapTag: string | null; +} + +export interface ToolSourceRow extends SourceRowBase { + name: string; + slug: string; + description: string | null; + categoryId: string | null; + locationId: string | null; + materials: string[]; + ppeRequired: string[]; + tags: string[]; + trainingRequired: boolean; + useRestrictions: string | null; + emergencyStop: string | null; + notes: string | null; + published: boolean; + archived: boolean; + /** ISO-8601 UTC, or null. */ + lastReviewedAt: string | null; + images: MirrorFile[]; +} + +export interface UnitSourceRow extends SourceRowBase { + /** Null when the unit has no tool, or its tool is archived. */ + toolId: string | null; + unitLabel: string; + serialNumber: string | null; + assetTag: string | null; + status: string; + condition: string | null; + /** `YYYY-MM-DD`, or null. */ + dateAcquired: string | null; + notes: string | null; +} + +export interface ResourceSourceRow extends SourceRowBase { + /** Null when the resource has no tool, or its tool is archived. */ + toolId: string | null; + title: string; + type: string | null; + url: string | null; + published: boolean; + notes: string | null; + files: MirrorFile[]; +} + +export interface MaintenanceSourceRow extends SourceRowBase { + title: string; + type: string | null; + priority: string | null; + status: string; + description: string | null; + resolution: string | null; + /** Null when the log names no tool, or its tool is archived. */ + toolId: string | null; + unitId: string | null; + toolName: string | null; + unitLabel: string | null; + reportedByName: string | null; + reportedByEmail: string | null; + assignedToName: string | null; + /** The assignee account's email. */ + assigneeEmail: string | null; + /** `YYYY-MM-DD`, or null. */ + dateReported: string | null; + dateResolved: string | null; +} + +export interface ProjectSourceRow extends SourceRowBase { + title: string; + link: string | null; + body: string; + materials: string[]; + /** The project's tools, archived ones left out. */ + toolIds: string[]; + authorName: string | null; + /** The author account's email. */ + authorEmail: string | null; + published: boolean; + /** ISO-8601 UTC, or null. */ + publishedAt: string | null; + photos: MirrorFile[]; +} + +export interface SourceRows { + categories: CategorySourceRow; + locations: LocationSourceRow; + tools: ToolSourceRow; + units: UnitSourceRow; + resources: ResourceSourceRow; + maintenance: MaintenanceSourceRow; + projects: ProjectSourceRow; +} + +export type AnySourceRow = SourceRows[MirrorEntity]; + +/** Where the previous page of rows ended: its last row's `revision` and `id`. */ +export interface SourceCursor { + revision: string; + id: string; +} + +export interface SourceQuery { + mirrorId: string; + /** `last_synced_at::text`, or null for every row. */ + since: string | null; + /** Continue after this row (keyset pagination on `updated_at, id`). */ + after?: SourceCursor | null; + /** Default 50. */ + limit?: number; + db?: Db; +} + +export const SOURCE_PAGE_SIZE = 50; + +// ── SQL pieces ────────────────────────────────────────────────────── + +/** A timestamptz as ISO-8601 UTC with milliseconds, computed by Postgres. */ +function iso(expression: SQL): SQL { + return sql`to_char((${expression}) at time zone 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"')`; +} + +/** The public files an owner has, as a JSON array of `{ url, name }` in position order. */ +function publicFiles(ownerType: "tool" | "resource" | "project"): SQL { + return sql`coalesce(( + select json_agg(json_build_object('url', a.public_url, 'name', a.original_filename) order by a.position, a.id) + from attachments a + where a.owner_type = ${ownerType} + and a.owner_id = s.id + and a.access = 'public' + and a.public_url is not null + ), '[]'::json)`; +} + +/** + * One page of `entity`'s changed rows. `columns` is the entity's select list + * (aliased columns of `s`), `joins` any extra joins, and `filter` the + * entity's own condition. Everything else — the page join, the two change + * tests, the cursor and the order — is the same for all seven. + */ +async function pageOf( + entity: MirrorEntity, + table: string, + query: SourceQuery, + parts: { columns: SQL; joins?: SQL; filter?: SQL; archive?: SQL } +): Promise { + const db = query.db ?? (await getDb()); + const limit = Math.max(1, Math.min(500, Math.trunc(query.limit ?? SOURCE_PAGE_SIZE))); + const conditions: SQL[] = [ + sql`(mp.entity_id is null or mp.source_updated_at is null or s.updated_at > mp.source_updated_at)`, + ]; + if (query.since !== null) conditions.push(sql`s.updated_at > ${query.since}::timestamptz`); + if (query.after) { + conditions.push(sql`(s.updated_at, s.id) > (${query.after.revision}::timestamptz, ${query.after.id}::uuid)`); + } + if (parts.filter) conditions.push(parts.filter); + + return rawRows( + db, + sql` + select s.id::text as "id", + s.updated_at::text as "revision", + ${iso(sql`s.updated_at`)} as "updatedAt", + mp.notion_page_id as "pageId", + ${parts.archive ?? sql`false`} as "archive", + ${parts.columns} + from ${sql.raw(`"${table}"`)} s + left join mirror_pages mp + on mp.mirror_id = ${query.mirrorId}::uuid + and mp.entity = ${entity} + and mp.entity_id = s.id + ${parts.joins ?? sql``} + where ${sql.join(conditions, sql` and `)} + order by s.updated_at, s.id + limit ${limit} + ` + ); +} + +/** A driver may hand json back as text; both drivers in use parse it, but a raw read is cheap to be sure of. */ +function json(value: unknown, fallback: T): T { + if (typeof value === "string") { + try { + return JSON.parse(value) as T; + } catch { + return fallback; + } + } + return (value ?? fallback) as T; +} + +function bool(value: unknown): boolean { + return value === true || value === "t" || value === "true"; +} + +function files(value: unknown): MirrorFile[] { + return json<{ url?: unknown; name?: unknown }[]>(value, []) + .filter((file) => typeof file?.url === "string" && file.url !== "") + .map((file) => ({ url: file.url as string, name: typeof file.name === "string" ? file.name : null })); +} + +function strings(value: unknown): string[] { + return json(value, []).filter((item): item is string => typeof item === "string"); +} + +/** The fields every row shares, typed. */ +function base(row: Record): SourceRowBase { + return { + id: String(row.id), + revision: String(row.revision), + updatedAt: String(row.updatedAt), + pageId: typeof row.pageId === "string" ? row.pageId : null, + archive: bool(row.archive), + }; +} + +function text(value: unknown): string | null { + return typeof value === "string" ? value : null; +} + +// ── Per entity ────────────────────────────────────────────────────── + +async function categoryRows(query: SourceQuery): Promise { + const rows = await pageOf>("categories", "categories", query, { + columns: sql`s.name as "name", s."group" as "group"`, + }); + return rows.map((row) => ({ ...base(row), name: String(row.name ?? ""), group: text(row.group) })); +} + +async function locationRows(query: SourceQuery): Promise { + const rows = await pageOf>("locations", "locations", query, { + columns: sql`s.room as "room", s.zone as "zone", s.map_tag as "mapTag"`, + }); + return rows.map((row) => ({ + ...base(row), + room: String(row.room ?? ""), + zone: String(row.zone ?? ""), + mapTag: text(row.mapTag), + })); +} + +async function toolRows(query: SourceQuery): Promise { + const rows = await pageOf>("tools", "tools", query, { + archive: sql`(s.archived_at is not null)`, + // An archived tool is only worth selecting when there is a page to archive. + filter: sql`(s.archived_at is null or mp.entity_id is not null)`, + columns: sql` + s.name as "name", + s.slug as "slug", + s.description as "description", + s.category_id::text as "categoryId", + s.location_id::text as "locationId", + to_json(s.materials) as "materials", + to_json(s.ppe_required) as "ppeRequired", + to_json(s.tags) as "tags", + s.training_required as "trainingRequired", + s.use_restrictions as "useRestrictions", + s.emergency_stop as "emergencyStop", + s.notes as "notes", + s.published as "published", + (s.archived_at is not null) as "archived", + ${iso(sql`s.last_reviewed_at`)} as "lastReviewedAt", + ${publicFiles("tool")} as "images"`, + }); + return rows.map((row) => ({ + ...base(row), + name: String(row.name ?? ""), + slug: String(row.slug ?? ""), + description: text(row.description), + categoryId: text(row.categoryId), + locationId: text(row.locationId), + materials: strings(row.materials), + ppeRequired: strings(row.ppeRequired), + tags: strings(row.tags), + trainingRequired: bool(row.trainingRequired), + useRestrictions: text(row.useRestrictions), + emergencyStop: text(row.emergencyStop), + notes: text(row.notes), + published: bool(row.published), + archived: bool(row.archived), + lastReviewedAt: text(row.lastReviewedAt), + images: files(row.images), + })); +} + +/** `tool_id`, or null when that tool is archived. */ +const LIVE_TOOL_ID = sql`(case when t.archived_at is null then s.tool_id::text else null end)`; +const JOIN_TOOL = sql`left join tools t on t.id = s.tool_id`; + +async function unitRows(query: SourceQuery): Promise { + const rows = await pageOf>("units", "units", query, { + joins: JOIN_TOOL, + columns: sql` + ${LIVE_TOOL_ID} as "toolId", + s.unit_label as "unitLabel", + s.serial_number as "serialNumber", + s.asset_tag as "assetTag", + s.status as "status", + s.condition as "condition", + s.date_acquired::text as "dateAcquired", + s.notes as "notes"`, + }); + return rows.map((row) => ({ + ...base(row), + toolId: text(row.toolId), + unitLabel: String(row.unitLabel ?? ""), + serialNumber: text(row.serialNumber), + assetTag: text(row.assetTag), + status: String(row.status ?? ""), + condition: text(row.condition), + dateAcquired: text(row.dateAcquired), + notes: text(row.notes), + })); +} + +async function resourceRows(query: SourceQuery): Promise { + const rows = await pageOf>("resources", "resources", query, { + joins: JOIN_TOOL, + columns: sql` + ${LIVE_TOOL_ID} as "toolId", + s.title as "title", + s.type as "type", + s.url as "url", + s.published as "published", + s.notes as "notes", + ${publicFiles("resource")} as "files"`, + }); + return rows.map((row) => ({ + ...base(row), + toolId: text(row.toolId), + title: String(row.title ?? ""), + type: text(row.type), + url: text(row.url), + published: bool(row.published), + notes: text(row.notes), + files: files(row.files), + })); +} + +async function maintenanceRows(query: SourceQuery): Promise { + const rows = await pageOf>("maintenance", "maintenance_logs", query, { + joins: sql`${JOIN_TOOL} left join "user" assignee on assignee.id = s.assigned_to_user_id`, + columns: sql` + s.title as "title", + s.type as "type", + s.priority as "priority", + s.status as "status", + s.description as "description", + s.resolution as "resolution", + ${LIVE_TOOL_ID} as "toolId", + s.unit_id::text as "unitId", + s.tool_name as "toolName", + s.unit_label as "unitLabel", + s.reported_by_name as "reportedByName", + s.reported_by_email as "reportedByEmail", + s.assigned_to_name as "assignedToName", + assignee.email as "assigneeEmail", + s.date_reported::text as "dateReported", + s.date_resolved::text as "dateResolved"`, + }); + return rows.map((row) => ({ + ...base(row), + title: String(row.title ?? ""), + type: text(row.type), + priority: text(row.priority), + status: String(row.status ?? ""), + description: text(row.description), + resolution: text(row.resolution), + toolId: text(row.toolId), + unitId: text(row.unitId), + toolName: text(row.toolName), + unitLabel: text(row.unitLabel), + reportedByName: text(row.reportedByName), + reportedByEmail: text(row.reportedByEmail), + assignedToName: text(row.assignedToName), + assigneeEmail: text(row.assigneeEmail), + dateReported: text(row.dateReported), + dateResolved: text(row.dateResolved), + })); +} + +async function projectRows(query: SourceQuery): Promise { + const rows = await pageOf>("projects", "projects", query, { + joins: sql`left join "user" author on author.id = s.author_user_id`, + archive: sql`(not s.published)`, + // Unpublished projects are never mirrored — unless one already was, and its page must go. + filter: sql`(s.published or mp.entity_id is not null)`, + columns: sql` + s.title as "title", + s.link as "link", + s.body as "body", + to_json(s.materials) as "materials", + coalesce(( + select json_agg(pt.tool_id::text order by pt.tool_id) + from project_tools pt + join tools pt_tool on pt_tool.id = pt.tool_id + where pt.project_id = s.id + and pt_tool.archived_at is null + ), '[]'::json) as "toolIds", + s.author_name as "authorName", + author.email as "authorEmail", + s.published as "published", + ${iso(sql`s.published_at`)} as "publishedAt", + ${publicFiles("project")} as "photos"`, + }); + return rows.map((row) => ({ + ...base(row), + title: String(row.title ?? ""), + link: text(row.link), + body: String(row.body ?? ""), + materials: strings(row.materials), + toolIds: strings(row.toolIds), + authorName: text(row.authorName), + authorEmail: text(row.authorEmail), + published: bool(row.published), + publishedAt: text(row.publishedAt), + photos: files(row.photos), + })); +} + +/** + * One page of `entity`'s rows that need pushing, oldest change first. Pass the + * last row back as `after` for the next page; an empty result means done. + */ +export async function listSourceRows(entity: E, query: SourceQuery): Promise { + switch (entity) { + case "categories": + return (await categoryRows(query)) as SourceRows[E][]; + case "locations": + return (await locationRows(query)) as SourceRows[E][]; + case "tools": + return (await toolRows(query)) as SourceRows[E][]; + case "units": + return (await unitRows(query)) as SourceRows[E][]; + case "resources": + return (await resourceRows(query)) as SourceRows[E][]; + case "maintenance": + return (await maintenanceRows(query)) as SourceRows[E][]; + case "projects": + return (await projectRows(query)) as SourceRows[E][]; + default: + return []; + } +} diff --git a/v5/src/lib/mirror/start.test.ts b/v5/src/lib/mirror/start.test.ts new file mode 100644 index 0000000..e2274d9 --- /dev/null +++ b/v5/src/lib/mirror/start.test.ts @@ -0,0 +1,149 @@ +// @vitest-environment node + +/** + * Starting the mirror's workflows, against PGlite (spec §3.8, §8 "Sync now: + * one push per mirror per 15 minutes"). + * + * `start` is mocked — the workflow tier has its own test — and so is the + * workflow module, so what is under test is the contract around the start: + * the database claim that enforces the fifteen minutes, the refusals, and + * that a start that fails gives the claim back. + */ + +const wf = vi.hoisted(() => ({ + start: vi.fn(), + mirrorPush: Object.assign(async () => ({}), { workflowId: "mirror-push" }), + mirrorPushAfterChange: Object.assign(async () => ({ mirrors: [] }), { workflowId: "mirror-push-after-change" }), +})); + +vi.mock("workflow/api", () => ({ start: wf.start })); +vi.mock("../../workflows/mirror-push", () => ({ + mirrorPush: wf.mirrorPush, + mirrorPushAfterChange: wf.mirrorPushAfterChange, +})); + +import { sql } from "drizzle-orm"; +import { getMirror, saveMirrorConnection, setMirrorMapping, setMirrorPaused, disconnectMirror } from "../data/mirrors"; +import { createPgliteDb } from "../db/pglite"; +import { notionMirrors, user } from "../db/schema/index"; +import type { Db } from "../db/types"; +import { startCoalescedPush, startMirrorPush, syncMirrorNow } from "./start"; + +const PAGE = "0f5e4a3c-1111-2222-3333-444455556666"; +const TOKEN_BYTES = new Uint8Array([1, 2, 3, 4]); + +let db: Db; +let owner: string; +let mirrorId: string; + +beforeAll(async () => { + db = await createPgliteDb(); +}); + +beforeEach(async () => { + wf.start.mockReset().mockResolvedValue({ runId: "run-1" }); + await db.delete(notionMirrors); + owner = `u-${crypto.randomUUID()}`; + await db.insert(user).values({ id: owner, name: "Mirror Owner", email: `${owner}@example.test`, role: "admin" }); + const { mirror } = await saveMirrorConnection( + { ownerUserId: owner, tokenCiphertext: TOKEN_BYTES, parentPageId: PAGE, parentPageTitle: "Mirror" }, + { db } + ); + mirrorId = mirror.id; + await setMirrorMapping(mirrorId, { tools: "b1a2c3d4-0000-4000-8000-000000000001" }, { db }); +}); + +describe("startMirrorPush", () => { + it("starts mirrorPush with the mirror id and answers the run id", async () => { + expect(await startMirrorPush(mirrorId)).toEqual({ ok: true, runId: "run-1" }); + expect(wf.start).toHaveBeenCalledWith(wf.mirrorPush, [mirrorId]); + }); + + it("answers ok: false, not a throw, when the workflow cannot be started", async () => { + vi.spyOn(console, "error").mockImplementation(() => {}); + wf.start.mockRejectedValue(new Error("world unavailable")); + expect(await startMirrorPush(mirrorId)).toEqual({ ok: false }); + }); +}); + +describe("syncMirrorNow", () => { + it("starts once, then refuses a second press within fifteen minutes with the seconds left", async () => { + expect(await syncMirrorNow(owner, { db })).toEqual({ ok: true }); + expect(wf.start).toHaveBeenCalledTimes(1); + expect(wf.start).toHaveBeenCalledWith(wf.mirrorPush, [mirrorId]); + + const second = await syncMirrorNow(owner, { db }); + + expect(second).toMatchObject({ ok: false, code: "sync_too_soon" }); + if (second.ok) throw new Error("unreachable"); + expect(second.retryAfterSeconds).toBeGreaterThan(14 * 60); + expect(second.retryAfterSeconds).toBeLessThanOrEqual(15 * 60); + expect(wf.start).toHaveBeenCalledTimes(1); + }); + + it("refuses as sync_running while another push holds the mirror, starts nothing, and keeps the window open", async () => { + await db.execute(sql`update notion_mirrors set running_since = now() where id = ${mirrorId}`); + + expect(await syncMirrorNow(owner, { db })).toEqual({ ok: false, code: "sync_running" }); + expect(wf.start).not.toHaveBeenCalled(); + expect((await getMirror(mirrorId, { db }))?.syncRequestedAt).toBeNull(); + + await db.execute(sql`update notion_mirrors set running_since = null where id = ${mirrorId}`); + expect(await syncMirrorNow(owner, { db })).toEqual({ ok: true }); + }); + + it("allows it again once the window has passed", async () => { + expect(await syncMirrorNow(owner, { db })).toEqual({ ok: true }); + await db.execute(sql`update notion_mirrors set sync_requested_at = now() - interval '16 minutes' where id = ${mirrorId}`); + expect(await syncMirrorNow(owner, { db })).toEqual({ ok: true }); + expect(wf.start).toHaveBeenCalledTimes(2); + }); + + it("gives the claim back when the start fails, so an immediate retry is allowed", async () => { + const error = vi.spyOn(console, "error").mockImplementation(() => {}); + wf.start.mockRejectedValueOnce(new Error("world unavailable")); + + expect(await syncMirrorNow(owner, { db })).toEqual({ ok: false, code: "start_failed" }); + expect((await getMirror(mirrorId, { db }))?.syncRequestedAt).toBeNull(); + expect(error).toHaveBeenCalled(); + + expect(await syncMirrorNow(owner, { db })).toEqual({ ok: true }); + expect(wf.start).toHaveBeenCalledTimes(2); + }); + + it("refuses a paused mirror, and starts nothing", async () => { + await setMirrorPaused(owner, true, { db }); + expect(await syncMirrorNow(owner, { db })).toEqual({ ok: false, code: "mirror_paused" }); + expect(wf.start).not.toHaveBeenCalled(); + }); + + it("refuses a disconnected mirror as not connected", async () => { + await disconnectMirror(owner, { db }); + expect(await syncMirrorNow(owner, { db })).toEqual({ ok: false, code: "not_connected" }); + expect(wf.start).not.toHaveBeenCalled(); + }); + + it("refuses somebody with no mirror at all as not connected", async () => { + expect(await syncMirrorNow("u-nobody", { db })).toEqual({ ok: false, code: "not_connected" }); + expect(wf.start).not.toHaveBeenCalled(); + }); + + it("refuses a mirror with nothing mapped", async () => { + await setMirrorMapping(mirrorId, {}, { db }); + expect(await syncMirrorNow(owner, { db })).toEqual({ ok: false, code: "not_mapped" }); + expect(wf.start).not.toHaveBeenCalled(); + }); +}); + +describe("startCoalescedPush", () => { + it("starts mirrorPushAfterChange with no arguments, so it sleeps the default delay", async () => { + expect(await startCoalescedPush()).toBe(true); + expect(wf.start).toHaveBeenCalledWith(wf.mirrorPushAfterChange, []); + }); + + it("answers false when the workflow cannot be started", async () => { + vi.spyOn(console, "error").mockImplementation(() => {}); + wf.start.mockRejectedValue(new Error("world unavailable")); + expect(await startCoalescedPush()).toBe(false); + }); +}); diff --git a/v5/src/lib/mirror/start.ts b/v5/src/lib/mirror/start.ts new file mode 100644 index 0000000..4c3af72 --- /dev/null +++ b/v5/src/lib/mirror/start.ts @@ -0,0 +1,111 @@ +import { start } from "workflow/api"; +import { mirrorPush, mirrorPushAfterChange } from "../../workflows/mirror-push.ts"; +import { claimManualSync, releaseManualSync } from "../data/mirrors.ts"; +import type { Db } from "../db/types.ts"; +import { scrubSecrets } from "./notion-client.ts"; + +/** + * Starting the mirror's workflows (spec §3.8 "Triggers", §8 rate limiting). + * + * The one module that imports `workflow/api` for the mirror. Everything that + * reaches it on an ordinary request does so through a dynamic `import()` in + * `trigger.ts`, taken only once a push has actually been claimed — so the + * inventory, intake and projects actions do not load the workflow runtime to + * find out that nobody has a mirror. + * + * **Starting is not pushing.** Each function answers whether the run was + * *started*; what the push did is on the mirror row once it has run, and + * `/admin/mirror` polls for it. + * + * Log lines carry the mirror id and a scrubbed reason, never a token or an + * email. + */ + +/** Start `mirrorPush(mirrorId)`. `{ ok: false }` when the workflow could not be started. */ +export async function startMirrorPush(mirrorId: string): Promise<{ ok: true; runId: string } | { ok: false }> { + try { + const run = await start(mirrorPush, [mirrorId]); + return { ok: true, runId: run.runId }; + } catch (error) { + console.error(`[mirror] could not start a push for mirror ${mirrorId}: ${reason(error)}`); + return { ok: false }; + } +} + +export type SyncNowResult = + | { ok: true } + | { + ok: false; + code: "not_connected" | "not_mapped" | "mirror_paused" | "sync_too_soon" | "sync_running" | "start_failed"; + retryAfterSeconds?: number; + }; + +/** + * **Sync now** (§3.8 control, §8: "one push per mirror per 15 minutes"). + * + * The limit lives in the database, not in this process: `claimManualSync` is + * one conditional update on `sync_requested_at`, so two presses from two tabs, + * or two server instances, cannot both pass it. A claim whose workflow then + * fails to start is given back (`releaseManualSync`), because a push that + * never ran must not spend the owner's fifteen minutes — they press again and + * it works. + * + * The mirror is always found from the owner: the caller passes the session's + * user id, never a mirror id from the page. A database failure throws; the + * server action turns it into its own `failed`. + */ +export async function syncMirrorNow(ownerUserId: string, options: { db?: Db } = {}): Promise { + const claim = await claimManualSync(ownerUserId, { db: options.db }); + if (!claim.ok) { + switch (claim.reason) { + case "not_found": + case "not_connected": + return { ok: false, code: "not_connected" }; + case "paused": + return { ok: false, code: "mirror_paused" }; + case "not_mapped": + return { ok: false, code: "not_mapped" }; + case "too_soon": + return { ok: false, code: "sync_too_soon", retryAfterSeconds: claim.retryAfterSeconds }; + case "running": + // Another push holds the mirror. Refused without spending the window: + // a push started now would be skipped, and the page would report the + // other push's result as this press's. + return { ok: false, code: "sync_running" }; + } + } + + const started = await startMirrorPush(claim.mirrorId); + if (started.ok) return { ok: true }; + + try { + await releaseManualSync(claim.mirrorId, { db: options.db }); + } catch { + // The claim stays, and the owner waits out the window. Still a failure to + // start, and still said so — never a success over a push that is not + // happening (Article 4). + console.error(`[mirror] could not release the Sync now claim on mirror ${claim.mirrorId}`); + } + return { ok: false, code: "start_failed" }; +} + +/** + * Start `mirrorPushAfterChange()` — the coalescing run (§3.8 trigger 1). False + * when it could not be started; `requestMirrorPush` then gives the claims back. + */ +export async function startCoalescedPush(): Promise { + try { + await start(mirrorPushAfterChange, []); + return true; + } catch (error) { + console.error(`[mirror] could not start the coalesced push: ${reason(error)}`); + return false; + } +} + +/** Why a start failed, in one short line with nothing token- or email-shaped in it. */ +function reason(error: unknown): string { + const message = error instanceof Error ? error.message || error.name : "unknown error"; + const oneLine = scrubSecrets(message).replace(/\s+/g, " ").trim(); + return oneLine.length > 200 ? `${oneLine.slice(0, 199)}…` : oneLine; +} diff --git a/v5/src/lib/mirror/steps.test.ts b/v5/src/lib/mirror/steps.test.ts new file mode 100644 index 0000000..23c00b1 --- /dev/null +++ b/v5/src/lib/mirror/steps.test.ts @@ -0,0 +1,130 @@ +// @vitest-environment node +import { FatalError, RetryableError } from "workflow"; + +/** + * The mirror's steps called directly (spec §10 "Workflow steps, called + * directly"). `pushMirror` is mocked — it has its own tests against the fake + * Notion — so what is under test is how a step treats a throw: the database + * having a bad minute is retried, anything else is fatal so a bug does not + * burn attempts, and nothing token- or email-shaped survives into the message. + */ + +const deps = vi.hoisted(() => ({ pushMirror: vi.fn(), takeCoalescedPush: vi.fn() })); + +vi.mock("./push", () => ({ pushMirror: deps.pushMirror })); +vi.mock("../data/mirrors", () => ({ takeCoalescedPush: deps.takeCoalescedPush })); + +import { DbUnavailableError } from "../db/client"; +import { MIRROR_STEP_MAX_RETRIES } from "./limits"; +import { finishMirrorPush, pushMirrorRound, takeCoalescedMirrors } from "./steps"; + +const MIRROR = "7a1c9a64-6a3b-4c8e-9d2f-0b1e2c3d4e5f"; + +beforeEach(() => { + deps.pushMirror.mockReset(); + deps.takeCoalescedPush.mockReset(); +}); + +async function thrownBy(promise: Promise): Promise { + try { + await promise; + } catch (error) { + return error; + } + throw new Error("expected the step to throw"); +} + +describe("pushMirrorRound", () => { + it("returns pushMirror's outcome unchanged", async () => { + deps.pushMirror.mockResolvedValue({ state: "incomplete", pushed: 12, archived: 0 }); + expect(await pushMirrorRound(MIRROR)).toEqual({ state: "incomplete", pushed: 12, archived: 0 }); + expect(deps.pushMirror).toHaveBeenCalledWith(MIRROR); + }); + + it(`carries maxRetries = ${MIRROR_STEP_MAX_RETRIES} as a property, where the SDK reads it`, () => { + expect(pushMirrorRound.maxRetries).toBe(MIRROR_STEP_MAX_RETRIES); + expect(takeCoalescedMirrors.maxRetries).toBe(MIRROR_STEP_MAX_RETRIES); + }); + + it.each<[string, unknown]>([ + ["DbUnavailableError", new DbUnavailableError(new Error("getaddrinfo ENOTFOUND"))], + ["a refused socket", Object.assign(new Error("connect ECONNREFUSED 127.0.0.1:5432"), { code: "ECONNREFUSED" })], + ["a dropped connection", new Error("Connection terminated unexpectedly")], + ["an admin shutdown (57P01)", Object.assign(new Error("terminating connection due to administrator command"), { code: "57P01" })], + ["a connection exception (08006)", Object.assign(new Error("connection failure"), { code: "08006" })], + ["a driver error wrapped by drizzle", new Error("Failed query: select 1", { cause: Object.assign(new Error("read ECONNRESET"), { code: "ECONNRESET" }) })], + ["Neon's HTTP driver failing to fetch", Object.assign(new Error("Error connecting to database"), { sourceError: new TypeError("fetch failed") })], + ])("retries %s a minute later", async (_label, error) => { + deps.pushMirror.mockRejectedValue(error); + + const thrown = await thrownBy(pushMirrorRound(MIRROR)); + + expect(RetryableError.is(thrown)).toBe(true); + expect((thrown as RetryableError).message).toBe("Mirror push: the database could not be reached."); + // "1m" — a Date one minute out, however the SDK normalises it. + const retryAfter = (thrown as RetryableError).retryAfter; + expect(retryAfter).toBeInstanceOf(Date); + const delta = (retryAfter as Date).getTime() - Date.now(); + expect(delta).toBeGreaterThan(50_000); + expect(delta).toBeLessThanOrEqual(60_000); + }); + + it.each<[string, unknown]>([ + ["a bug", new TypeError("Cannot read properties of undefined (reading 'id')")], + ["a constraint violation", Object.assign(new Error('violates check constraint "mirror_pages_entity_check"'), { code: "23514" })], + ["a thrown string", "nope"], + ])("fails %s at once, so a bug does not burn retries", async (_label, error) => { + deps.pushMirror.mockRejectedValue(error); + + const thrown = await thrownBy(pushMirrorRound(MIRROR)); + + expect(FatalError.is(thrown)).toBe(true); + expect((thrown as Error).message).toMatch(/^Mirror push: /); + }); + + it("scrubs anything token- or email-shaped out of a fatal message", async () => { + deps.pushMirror.mockRejectedValue( + new Error("Failed query: insert … params: ntn_abcdefghijklmnop1234, casey@cornell.edu, secret_ZYXWVUT98765") + ); + + const message = (await thrownBy(pushMirrorRound(MIRROR)) as Error).message; + + expect(message).not.toContain("ntn_abcdefghijklmnop1234"); + expect(message).not.toContain("secret_ZYXWVUT98765"); + expect(message).not.toContain("casey@cornell.edu"); + }); + + it("passes an already-classified error through", async () => { + const fatal = new FatalError("already decided"); + deps.pushMirror.mockRejectedValue(fatal); + expect(await thrownBy(pushMirrorRound(MIRROR))).toBe(fatal); + }); +}); + +describe("takeCoalescedMirrors", () => { + it("returns the ids takeCoalescedPush cleared", async () => { + deps.takeCoalescedPush.mockResolvedValue([MIRROR]); + expect(await takeCoalescedMirrors()).toEqual([MIRROR]); + }); + + it("retries the database being unreachable", async () => { + deps.takeCoalescedPush.mockRejectedValue(new DbUnavailableError(new Error("down"))); + expect(RetryableError.is(await thrownBy(takeCoalescedMirrors()))).toBe(true); + }); +}); + +describe("finishMirrorPush", () => { + it("logs one line of ids and counts, and nothing else", async () => { + const info = vi.spyOn(console, "info").mockImplementation(() => {}); + + await finishMirrorPush("change", [ + { mirrorId: MIRROR, rounds: 2, state: "ok", pushed: 41, archived: 1, failed: 0 }, + { mirrorId: "m2", rounds: 1, state: "partial", pushed: 3, archived: 0, failed: 2 }, + ]); + + expect(info).toHaveBeenCalledTimes(1); + expect(info.mock.calls[0]).toEqual([ + `[mirror] change push finished: mirrors=2; ${MIRROR} rounds=2 state=ok pushed=41 archived=1 failed=0; m2 rounds=1 state=partial pushed=3 archived=0 failed=2`, + ]); + }); +}); diff --git a/v5/src/lib/mirror/steps.ts b/v5/src/lib/mirror/steps.ts new file mode 100644 index 0000000..acc7aa5 --- /dev/null +++ b/v5/src/lib/mirror/steps.ts @@ -0,0 +1,188 @@ +import { FatalError, RetryableError } from "workflow"; +import { takeCoalescedPush } from "../data/mirrors.ts"; +import { MIRROR_STEP_MAX_RETRIES } from "./limits.ts"; +import { scrubSecrets } from "./notion-client.ts"; +import { pushMirror, type MirrorPushOutcome } from "./push.ts"; + +/** + * The Notion mirror's workflow steps (spec §3.8 "Push" and "Triggers", the + * 2026-09-22 amendment's Workflow SDK rules). + * + * Three steps, each short: + * + * 1. {@link pushMirrorRound} — one call to `pushMirror`, which claims the + * mirror, pushes for at most 45 seconds (`MIRROR_PUSH_BUDGET_MS`) and + * records what happened. Far inside the Hobby plan's 300-second function + * ceiling; a first sync bigger than one budget comes back `incomplete` and + * the workflow runs another round. + * 2. {@link takeCoalescedMirrors} — the coalescing run woke up: clear the + * waiting claims and return the mirrors to push. + * 3. {@link finishMirrorPush} — one log line with counts and mirror ids. + * + * **Retries are for the database, and only for its bad minute.** `pushMirror` + * turns every Notion failure into an outcome — a status and an error on the + * mirror row — and throws only when Postgres does, having already released + * `running_since` in a `finally`. So a throw here is classified once: an + * unreachable database or a dropped connection is a {@link RetryableError} + * (retried a minute later, `maxRetries` = {@link MIRROR_STEP_MAX_RETRIES}); + * anything else is a {@link FatalError}, because a bug gives the same answer + * every time and should not burn attempts. The message is scrubbed of anything + * token- or email-shaped before it reaches the run's event log. + * + * `maxRetries` is set **as a property on each step function**, which is how + * the Workflow SDK reads it. + * + * Steps run from a pre-built bundle under plain Node (`@workflow/vitest` + * locally, the step route in production), so nothing here or below it may + * import `"server-only"` or `next/*`, and every import is relative. + */ + +/** What one mirror's rounds came to, for the log line and the workflow's return value. */ +export interface MirrorPushSummary { + mirrorId: string; + /** How many times `pushMirror` was called. */ + rounds: number; + /** The last round's state, or `error` when a round threw after its retries. */ + state: MirrorPushOutcome["state"] | "error"; + pushed: number; + archived: number; + failed: number; +} + +/** Which door the push came in through, for the log line. */ +export type MirrorPushTrigger = "run" | "change"; + +/** A step error's own reason is kept, but only this much of it. */ +const MAX_DETAIL_LENGTH = 200; + +/** How long a step waits before retrying after the database dropped out. */ +const DB_RETRY_AFTER = "1m"; + +/** + * Push one mirror once (§3.8 steps 1–5). Returns the outcome unchanged; the + * workflow decides whether to go again. + */ +export async function pushMirrorRound(mirrorId: string): Promise { + "use step"; + try { + return await pushMirror(mirrorId); + } catch (error) { + throw classifyMirrorStepError(error, "push"); + } +} +pushMirrorRound.maxRetries = MIRROR_STEP_MAX_RETRIES; + +/** + * Clear every waiting coalesced-push claim and return the ids of the active + * mirrors among them (§3.8 trigger 1: "pushes every active mirror"). + */ +export async function takeCoalescedMirrors(): Promise { + "use step"; + try { + return await takeCoalescedPush(); + } catch (error) { + throw classifyMirrorStepError(error, "take"); + } +} +takeCoalescedMirrors.maxRetries = MIRROR_STEP_MAX_RETRIES; + +/** + * The run is done. **One line**: the trigger, then each mirror's id, rounds, + * final state and counts. No page titles, no row content, no people, and + * never a token — the outcome's error detail is deliberately left out; it is + * on the mirror row, where `/admin/mirror` shows it. + */ +export async function finishMirrorPush(trigger: MirrorPushTrigger, summaries: MirrorPushSummary[]): Promise { + "use step"; + const parts = summaries.map( + (s) => + `${s.mirrorId} rounds=${s.rounds} state=${s.state} pushed=${s.pushed} archived=${s.archived} failed=${s.failed}` + ); + console.info( + `[mirror] ${trigger} push finished: mirrors=${summaries.length}${parts.length > 0 ? `; ${parts.join("; ")}` : ""}` + ); +} + +// ── Error classification ──────────────────────────────────────────── + +/** Node's socket-level failures: the database was not there to answer. */ +const CONNECTION_CODES = new Set([ + "ECONNREFUSED", + "ECONNRESET", + "ECONNABORTED", + "ETIMEDOUT", + "EPIPE", + "ENOTFOUND", + "EAI_AGAIN", + "EHOSTUNREACH", + "ENETUNREACH", + "UND_ERR_CONNECT_TIMEOUT", + "UND_ERR_SOCKET", +]); + +/** + * Postgres SQLSTATEs that mean "try again": class 08 (connection exception), + * 57P01–57P03 (the server shutting down or not accepting connections yet), + * 53300 (too many connections), and the two transaction conflicts a retry + * exists for (serialization failure, deadlock). + */ +function isTransientSqlState(code: string): boolean { + return ( + /^08[0-9A-Z]{3}$/.test(code) || + code === "57P01" || + code === "57P02" || + code === "57P03" || + code === "53300" || + code === "40001" || + code === "40P01" + ); +} + +const CONNECTION_MESSAGE = + /connection (terminated|refused|reset|timed out|closed|ended)|terminating connection|fetch failed|socket hang up|database is unavailable/i; + +/** + * True when `error`, or anything in its `cause` chain (drizzle wraps the + * driver's error; Neon's HTTP driver wraps `fetch`'s), is the database being + * unreachable rather than the query being wrong. Read by shape, not + * `instanceof`: the error may come from another bundle's copy of the class. + */ +export function isTransientDbError(error: unknown): boolean { + let current: unknown = error; + for (let depth = 0; depth < 6 && typeof current === "object" && current !== null; depth += 1) { + const { name, code, message, cause, sourceError } = current as { + name?: unknown; + code?: unknown; + message?: unknown; + cause?: unknown; + sourceError?: unknown; + }; + if (name === "DbUnavailableError") return true; + if (typeof code === "string" && (CONNECTION_CODES.has(code) || isTransientSqlState(code))) return true; + if (typeof message === "string" && CONNECTION_MESSAGE.test(message)) return true; + current = cause ?? sourceError; + } + return false; +} + +/** The error a step should throw. Already-classified errors pass through. */ +function classifyMirrorStepError(error: unknown, stage: "push" | "take"): FatalError | RetryableError { + if (FatalError.is(error) || RetryableError.is(error)) return error; + const label = stage === "push" ? "Mirror push" : "Mirror push (taking the waiting pushes)"; + if (isTransientDbError(error)) { + return new RetryableError(`${label}: the database could not be reached.`, { retryAfter: DB_RETRY_AFTER }); + } + return new FatalError(`${label}: ${detail(error)}`); +} + +/** One line, short, with anything token- or email-shaped removed. */ +function detail(error: unknown): string { + const raw = + typeof error === "object" && error !== null && typeof (error as { message?: unknown }).message === "string" + ? (error as { message: string }).message || "unknown error" + : typeof error === "string" + ? error + : "unknown error"; + const oneLine = scrubSecrets(raw).replace(/\s+/g, " ").trim(); + return oneLine.length > MAX_DETAIL_LENGTH ? `${oneLine.slice(0, MAX_DETAIL_LENGTH - 1)}…` : oneLine; +} diff --git a/v5/src/lib/mirror/token-crypto.test.ts b/v5/src/lib/mirror/token-crypto.test.ts new file mode 100644 index 0000000..fb31c66 --- /dev/null +++ b/v5/src/lib/mirror/token-crypto.test.ts @@ -0,0 +1,113 @@ +// @vitest-environment node +import { + MirrorKeyUnavailableError, + MirrorTokenUnreadableError, + decryptMirrorToken, + encryptMirrorToken, + mirrorKeyAvailable, +} from "./token-crypto"; + +/** + * The mirror token at rest (spec §8, §10 "Token encryption: round-trips; a + * different key fails to decrypt"). Nothing here reads a real secret: every + * secret is a test string, passed explicitly or through `vi.stubEnv`. + */ +const TOKEN = "ntn_TESTtoken0123456789abcdefABCDEF"; +const SECRET = "test-auth-secret-for-mirror-crypto"; + +function errorOf(fn: () => unknown): Error { + try { + fn(); + } catch (error) { + return error as Error; + } + throw new Error("expected a throw"); +} + +describe("mirror token encryption", () => { + it("round-trips", () => { + const stored = encryptMirrorToken(TOKEN, SECRET); + expect(stored).toBeInstanceOf(Uint8Array); + expect(stored[0]).toBe(0x01); + expect(stored.length).toBe(1 + 12 + 16 + Buffer.byteLength(TOKEN)); + expect(decryptMirrorToken(stored, SECRET)).toBe(TOKEN); + }); + + it("reads AUTH_SECRET from the environment by default", () => { + vi.stubEnv("AUTH_SECRET", SECRET); + const stored = encryptMirrorToken(TOKEN); + expect(decryptMirrorToken(stored)).toBe(TOKEN); + expect(decryptMirrorToken(stored, SECRET)).toBe(TOKEN); + }); + + it("fails to decrypt under a different AUTH_SECRET", () => { + vi.stubEnv("AUTH_SECRET", SECRET); + const stored = encryptMirrorToken(TOKEN); + + vi.stubEnv("AUTH_SECRET", "a-rotated-auth-secret"); + expect(() => decryptMirrorToken(stored)).toThrow(MirrorTokenUnreadableError); + }); + + it("fails on a tampered byte anywhere", () => { + const stored = encryptMirrorToken(TOKEN, SECRET); + for (const index of [5, 20, stored.length - 1]) { + const tampered = new Uint8Array(stored); + tampered[index] ^= 0x01; + expect(() => decryptMirrorToken(tampered, SECRET)).toThrow(MirrorTokenUnreadableError); + } + }); + + it("fails on a truncated value or an unknown version", () => { + const stored = encryptMirrorToken(TOKEN, SECRET); + expect(() => decryptMirrorToken(stored.subarray(0, 29), SECRET)).toThrow(MirrorTokenUnreadableError); + const versioned = new Uint8Array(stored); + versioned[0] = 0x02; + expect(() => decryptMirrorToken(versioned, SECRET)).toThrow(MirrorTokenUnreadableError); + }); + + it("never encrypts the same token to the same bytes", () => { + const a = encryptMirrorToken(TOKEN, SECRET); + const b = encryptMirrorToken(TOKEN, SECRET); + expect(Buffer.from(a).equals(Buffer.from(b))).toBe(false); + }); + + it("does not contain the plaintext", () => { + const stored = Buffer.from(encryptMirrorToken(TOKEN, SECRET)); + expect(stored.includes(Buffer.from(TOKEN))).toBe(false); + expect(stored.includes(Buffer.from(TOKEN.slice(0, 12)))).toBe(false); + }); + + it("refuses an empty secret as an unavailable key", () => { + expect(() => encryptMirrorToken(TOKEN, "")).toThrow(MirrorKeyUnavailableError); + expect(() => decryptMirrorToken(encryptMirrorToken(TOKEN, SECRET), "")).toThrow(MirrorKeyUnavailableError); + vi.stubEnv("AUTH_SECRET", ""); + expect(() => encryptMirrorToken(TOKEN)).toThrow(MirrorKeyUnavailableError); + }); + + it("says whether a key is available", () => { + expect(mirrorKeyAvailable(SECRET)).toBe(true); + expect(mirrorKeyAvailable("")).toBe(false); + expect(mirrorKeyAvailable(" ")).toBe(false); + vi.stubEnv("AUTH_SECRET", ""); + expect(mirrorKeyAvailable()).toBe(false); + vi.stubEnv("AUTH_SECRET", SECRET); + expect(mirrorKeyAvailable()).toBe(true); + }); + + it("puts neither the token nor the secret into any error", () => { + const stored = encryptMirrorToken(TOKEN, SECRET); + const tampered = new Uint8Array(stored); + tampered[stored.length - 1] ^= 0xff; + const errors = [ + errorOf(() => decryptMirrorToken(stored, "another-secret")), + errorOf(() => decryptMirrorToken(tampered, SECRET)), + errorOf(() => encryptMirrorToken(TOKEN, "")), + ]; + for (const error of errors) { + const text = `${error.name} ${error.message} ${error.stack ?? ""} ${String(error.cause ?? "")}`; + expect(text).not.toContain(TOKEN); + expect(text).not.toContain(SECRET); + expect(error.cause).toBeUndefined(); + } + }); +}); diff --git a/v5/src/lib/mirror/token-crypto.ts b/v5/src/lib/mirror/token-crypto.ts new file mode 100644 index 0000000..8b9f6d8 --- /dev/null +++ b/v5/src/lib/mirror/token-crypto.ts @@ -0,0 +1,99 @@ +import { createCipheriv, createDecipheriv, hkdfSync, randomBytes } from "node:crypto"; + +/** + * A mirror's Notion token at rest (spec §8 "Secrets at rest", open question 5 + * as answered on 2026-09-23). + * + * AES-256-GCM under a key derived from `AUTH_SECRET` with HKDF-SHA256, a fixed + * salt and a fixed info string, so no new environment variable is needed. The + * trade is named in the spec: rotating `AUTH_SECRET` ends every session **and** + * makes every stored token unreadable, and the mirror page then asks for the + * token again. + * + * Layout of the stored value: `0x01 | iv (12) | tag (16) | ciphertext`. The + * leading version byte leaves room for a different key or cipher later without + * guessing at what an old row holds. + * + * **No error thrown here carries the token, the secret or the key** — not in + * its message and not as a `cause` — because an error message is exactly the + * thing that ends up in a log line (§10: "a mirror token shows up in a log line + * or an error message"). + * + * Plain Node (`node:crypto`), no `server-only`, no `@/` alias: workflow step + * code decrypts the token from an esbuild bundle. + */ + +export const MIRROR_TOKEN_KEY_INFO = "makerlab-tools/notion-mirror-token/v1"; + +const SALT = "makerlab-tools-mirror"; +const VERSION = 0x01; +const IV_BYTES = 12; +const TAG_BYTES = 16; +const HEADER_BYTES = 1 + IV_BYTES + TAG_BYTES; + +/** `AUTH_SECRET` is unset or empty, so no key can be derived. */ +export class MirrorKeyUnavailableError extends Error { + constructor() { + super("The mirror token key is unavailable: AUTH_SECRET is not set."); + this.name = "MirrorKeyUnavailableError"; + } +} + +/** The stored value does not decrypt under the current key — rotated secret, or tampered bytes. */ +export class MirrorTokenUnreadableError extends Error { + constructor() { + super("The stored mirror token cannot be decrypted with the current AUTH_SECRET."); + this.name = "MirrorTokenUnreadableError"; + } +} + +/** True when a key can be derived — `AUTH_SECRET` (or `secret`) is set and not blank. */ +export function mirrorKeyAvailable(secret: string | undefined = process.env.AUTH_SECRET): boolean { + return typeof secret === "string" && secret.trim().length > 0; +} + +function deriveKey(secret: string): Buffer { + if (!mirrorKeyAvailable(secret)) throw new MirrorKeyUnavailableError(); + return Buffer.from(hkdfSync("sha256", secret, SALT, MIRROR_TOKEN_KEY_INFO, 32)); +} + +/** Encrypt `token` for `notion_mirrors.token_ciphertext`. Two calls never produce the same bytes. */ +export function encryptMirrorToken(token: string, secret: string = process.env.AUTH_SECRET ?? ""): Uint8Array { + const key = deriveKey(secret); + const iv = randomBytes(IV_BYTES); + const cipher = createCipheriv("aes-256-gcm", key, iv); + const ciphertext = Buffer.concat([cipher.update(token, "utf8"), cipher.final()]); + const tag = cipher.getAuthTag(); + return new Uint8Array(Buffer.concat([Buffer.from([VERSION]), iv, tag, ciphertext])); +} + +/** + * Decrypt a stored token. + * + * @throws MirrorKeyUnavailableError when there is no secret to derive a key from. + * @throws MirrorTokenUnreadableError for a wrong key, a tampered or truncated + * value, or an unknown version byte. Deliberately the same error for all of + * them: the remedy is the same (connect again), and the difference is not + * something to show anybody. + */ +export function decryptMirrorToken( + ciphertext: Uint8Array, + secret: string = process.env.AUTH_SECRET ?? "" +): string { + const key = deriveKey(secret); + const bytes = Buffer.from(ciphertext.buffer, ciphertext.byteOffset, ciphertext.byteLength); + if (bytes.length <= HEADER_BYTES || bytes[0] !== VERSION) throw new MirrorTokenUnreadableError(); + + const iv = bytes.subarray(1, 1 + IV_BYTES); + const tag = bytes.subarray(1 + IV_BYTES, HEADER_BYTES); + const body = bytes.subarray(HEADER_BYTES); + try { + const decipher = createDecipheriv("aes-256-gcm", key, iv); + decipher.setAuthTag(tag); + return Buffer.concat([decipher.update(body), decipher.final()]).toString("utf8"); + } catch { + // Node's own message ("Unsupported state or unable to authenticate data") + // says nothing secret, but nothing is gained by passing it on either. + throw new MirrorTokenUnreadableError(); + } +} diff --git a/v5/src/lib/mirror/trigger.test.ts b/v5/src/lib/mirror/trigger.test.ts new file mode 100644 index 0000000..91e7321 --- /dev/null +++ b/v5/src/lib/mirror/trigger.test.ts @@ -0,0 +1,136 @@ +// @vitest-environment node + +/** + * `requestMirrorPush()` against PGlite (spec §3.8 trigger 1). + * + * `start.ts` is mocked with a factory that counts how often it is loaded, so + * the first test can prove the path every deployment without a mirror takes: + * one query, and the workflow runtime never loaded. The rest prove the + * coalescing — a burst of changes starts one run — and that nothing here ever + * throws into the write that called it. + */ + +const starter = vi.hoisted(() => ({ loads: 0, startCoalescedPush: vi.fn() })); + +vi.mock("./start", () => { + starter.loads += 1; + return { startCoalescedPush: starter.startCoalescedPush }; +}); + +import { sql } from "drizzle-orm"; +import { getMirror, saveMirrorConnection, setMirrorPaused } from "../data/mirrors"; +import { createPgliteDb } from "../db/pglite"; +import { notionMirrors, user } from "../db/schema/index"; +import type { Db } from "../db/types"; +import { requestMirrorPush } from "./trigger"; + +const PAGE = "0f5e4a3c-1111-2222-3333-444455556666"; + +let db: Db; + +beforeAll(async () => { + db = await createPgliteDb(); +}); + +beforeEach(async () => { + starter.startCoalescedPush.mockReset().mockResolvedValue(true); + await db.delete(notionMirrors); +}); + +async function mirror(): Promise<{ owner: string; id: string }> { + const owner = `u-${crypto.randomUUID()}`; + await db.insert(user).values({ id: owner, name: "Mirror Owner", email: `${owner}@example.test`, role: "admin" }); + const { mirror: created } = await saveMirrorConnection( + { ownerUserId: owner, tokenCiphertext: new Uint8Array([1, 2, 3]), parentPageId: PAGE, parentPageTitle: null }, + { db } + ); + return { owner, id: created.id }; +} + +describe("requestMirrorPush", () => { + // Declared first on purpose: the module is loaded at most once per file. + it("with no active mirror, neither loads start.ts nor starts anything", async () => { + await requestMirrorPush({ db }); + + const { owner } = await mirror(); + await setMirrorPaused(owner, true, { db }); + await requestMirrorPush({ db }); + + expect(starter.loads).toBe(0); + expect(starter.startCoalescedPush).not.toHaveBeenCalled(); + }); + + it("with an active mirror, starts one coalesced push and leaves it claimed", async () => { + const { id } = await mirror(); + + await requestMirrorPush({ db }); + + expect(starter.startCoalescedPush).toHaveBeenCalledTimes(1); + expect((await getMirror(id, { db }))?.pushRequestedAt).not.toBeNull(); + }); + + it("coalesces a burst: five changes start one run", async () => { + await mirror(); + + for (let i = 0; i < 5; i += 1) await requestMirrorPush({ db }); + + expect(starter.startCoalescedPush).toHaveBeenCalledTimes(1); + }); + + it("coalesces five changes that arrive at once", async () => { + await mirror(); + + await Promise.all(Array.from({ length: 5 }, () => requestMirrorPush({ db }))); + + expect(starter.startCoalescedPush).toHaveBeenCalledTimes(1); + }); + + it("claims again once the waiting push is older than the stale window", async () => { + const { id } = await mirror(); + await requestMirrorPush({ db }); + await db.execute(sql`update notion_mirrors set push_requested_at = now() - interval '11 minutes' where id = ${id}`); + + await requestMirrorPush({ db }); + + expect(starter.startCoalescedPush).toHaveBeenCalledTimes(2); + }); + + it("gives the claim back when the start fails, so the next change tries again", async () => { + const { id } = await mirror(); + starter.startCoalescedPush.mockResolvedValueOnce(false); + + await requestMirrorPush({ db }); + expect((await getMirror(id, { db }))?.pushRequestedAt).toBeNull(); + + await requestMirrorPush({ db }); + expect(starter.startCoalescedPush).toHaveBeenCalledTimes(2); + expect((await getMirror(id, { db }))?.pushRequestedAt).not.toBeNull(); + }); + + it("gives the claim back when starting throws, and does not throw itself", async () => { + const { id } = await mirror(); + const error = vi.spyOn(console, "error").mockImplementation(() => {}); + starter.startCoalescedPush.mockRejectedValueOnce(new Error("world unavailable")); + + await expect(requestMirrorPush({ db })).resolves.toBeUndefined(); + + expect((await getMirror(id, { db }))?.pushRequestedAt).toBeNull(); + expect(error).toHaveBeenCalledTimes(1); + }); + + it("swallows a database error with one fixed line that names nothing", async () => { + const error = vi.spyOn(console, "error").mockImplementation(() => {}); + const broken = Object.create(db) as Db; + broken.update = (() => { + throw new Error("connection terminated unexpectedly; owner casey@cornell.edu"); + }) as unknown as Db["update"]; + + await expect(requestMirrorPush({ db: broken })).resolves.toBeUndefined(); + + expect(starter.startCoalescedPush).not.toHaveBeenCalled(); + expect(error).toHaveBeenCalledTimes(1); + expect(error.mock.calls[0]).toEqual([ + "[mirror] could not request a mirror push; the next change or the daily backstop will retry", + ]); + }); +}); diff --git a/v5/src/lib/mirror/trigger.ts b/v5/src/lib/mirror/trigger.ts new file mode 100644 index 0000000..747c407 --- /dev/null +++ b/v5/src/lib/mirror/trigger.ts @@ -0,0 +1,51 @@ +import { claimCoalescedPush, releaseCoalescedPush } from "../data/mirrors.ts"; +import type { Db } from "../db/types.ts"; + +/** + * `requestMirrorPush()` — "something the mirror carries just changed" (spec + * §3.8 trigger 1). + * + * Called after a committed write by approving a tool (`intake/approve.ts`), + * every tool-editor write that landed (`admin/inventory/tool-write-context.ts` + * — saves, publish, archive, Looks good, units, resources, photos), + * publishing or unpublishing a project (`admin/projects/actions.ts`), and + * working a maintenance ticket (`admin/maintenance/actions.ts`). Never after a + * refused write: nothing changed. Writes nobody wires here (a student's + * `report_issue`, category or location edits outside the editor) catch up on + * the daily backstop. + * + * **It never throws, and it never makes the caller wait on Notion.** A failed + * push never affects the app (§3.8, Goal 8): the write has already committed, + * and a mirror that could not be told about it catches up on the next change + * or on the daily backstop. So every failure here is caught and leaves one + * fixed log line with nothing about the write or the person in it. + * + * **One query when there is nothing to do.** `claimCoalescedPush` marks every + * active mirror that has no push already waiting and returns their ids. None — + * no mirror at all, or a push already on its way — is the common path, and it + * starts nothing. Only a non-empty claim loads `start.ts` (with a dynamic + * `import()`, so `workflow/api` stays out of this module's static graph and + * out of every action test that reaches it) and starts `mirrorPushAfterChange`, + * which sleeps two minutes so a burst of edits becomes one push. A start that + * fails gives the claims back, so the next change tries again rather than + * waiting out the ten-minute stale window. + */ +export async function requestMirrorPush(options: { db?: Db } = {}): Promise { + let claimed: string[] = []; + try { + claimed = await claimCoalescedPush({ db: options.db }); + if (claimed.length === 0) return; + + const { startCoalescedPush } = await import("./start.ts"); + if (await startCoalescedPush()) return; + } catch { + console.error("[mirror] could not request a mirror push; the next change or the daily backstop will retry"); + } + + if (claimed.length === 0) return; + try { + await releaseCoalescedPush(claimed, { db: options.db }); + } catch { + console.error("[mirror] could not release a coalesced push claim; it expires on its own"); + } +} diff --git a/v5/src/lib/mirror/types.ts b/v5/src/lib/mirror/types.ts new file mode 100644 index 0000000..c7ee43f --- /dev/null +++ b/v5/src/lib/mirror/types.ts @@ -0,0 +1,106 @@ +import type { MirrorEntity, MirrorStatus } from "../db/schema/vocabulary.ts"; + +/** + * Client-safe shapes for the Notion mirror (spec §3.8, §4.12). + * + * This module imports only the vocabulary, so a client component + * (`MirrorStatus`, `MirrorMapping`) can import it without pulling the database, + * the crypto or the Notion client into the browser bundle. Relative imports + * with `.ts` extensions: workflow step code reaches it under plain Node. + */ + +/** Notion database ids by entity, dashed lower-case. An entity with no id is not pushed. */ +export type MirrorMapping = Partial>; + +/** + * Why the last push did not finish cleanly — `notion_mirrors.last_error.code`. + * Every code has an `admin.mirror.lastError.` string. + */ +export const MIRROR_ERROR_CODES = [ + "unauthorized", + "token_unreadable", + "key_unavailable", + "database_not_found", + "schema_mismatch", + "rows_failed", + "budget_exhausted", + "notion_unavailable", + "unknown", +] as const; +export type MirrorErrorCode = (typeof MIRROR_ERROR_CODES)[number]; + +/** + * `notion_mirrors.last_error`: a code the page translates, which entities it + * concerned, how many rows failed, and a short English diagnosis. + * + * `detail` is scrubbed of anything token-shaped, at most 300 characters, and + * never holds a token or an email — it is shown on the page and may be copied + * into an issue. + */ +export interface MirrorLastError { + code: MirrorErrorCode; + entities: MirrorEntity[]; + failed: number; + detail: string | null; +} + +/** + * Why a setup call (test, connect, create databases, save mapping, sync now) + * was refused. Every code has an `admin.mirror.errors.` string. + */ +export const MIRROR_SETUP_ERRORS = [ + "invalid_token", + "invalid_page", + "invalid_database_id", + "unauthorized", + "page_not_found", + "database_not_found", + "schema_mismatch", + "notion_unavailable", + "key_unavailable", + "token_unreadable", + "not_connected", + "not_mapped", + "mirror_paused", + "sync_too_soon", + "sync_running", + "start_failed", +] as const; +export type MirrorSetupError = (typeof MIRROR_SETUP_ERRORS)[number]; + +/** One pasted database id that did not validate, and why. */ +export interface MappingProblem { + entity: MirrorEntity; + code: "invalid_database_id" | "database_not_found" | "schema_mismatch"; + /** Expected property names the database lacks. */ + missing?: string[]; + /** Expected property names present with the wrong type. */ + wrongType?: string[]; +} + +/** + * What `/admin/mirror` renders — the owner's mirror, never its token. Every + * date is an ISO string, and every flag is computed by Postgres against its own + * `now()`, so the page and the claim that will refuse a second Sync now agree. + */ +export interface MirrorView { + id: string; + /** A token is stored. Disconnect forgets it and keeps everything else. */ + connected: boolean; + parentPageId: string; + parentPageTitle: string | null; + mapping: MirrorMapping; + paused: boolean; + /** `running_since` is set and newer than 15 minutes. */ + running: boolean; + /** `sync_requested_at` is set and `last_run_at` is null or older than it. */ + syncPending: boolean; + /** `push_requested_at` is set: a change is waiting for the coalesced push. */ + pushScheduled: boolean; + lastSyncedAt: string | null; + lastRunAt: string | null; + lastStatus: MirrorStatus | null; + lastError: MirrorLastError | null; + /** When Sync now is next allowed; null means it is allowed now. */ + syncAvailableAt: string | null; +} diff --git a/v5/src/lib/rate-limit.ts b/v5/src/lib/rate-limit.ts index 941e050..5a64d1e 100644 --- a/v5/src/lib/rate-limit.ts +++ b/v5/src/lib/rate-limit.ts @@ -162,6 +162,16 @@ export function chatTierFor(role: Role): RateLimitTier { */ export const ADMIN_ACTION_TIER: RateLimitTier = { limit: 120, windowMs: 60_000 }; +/** + * Notion mirror setup calls per minute, per admin (spec §8), keyed + * `mirror-setup:`: test connection, connect, create databases and + * save mapping. Each one calls Notion with the admin's own token, so this + * bounds how hard a script holding a session could drive that token against + * Notion's rate limit — on top of `ADMIN_ACTION_TIER`, which every action + * passes as well. + */ +export const MIRROR_SETUP_TIER: RateLimitTier = { limit: 10, windowMs: 60_000 }; + /** * Limits for the non-chat routes — unchanged from before sign-in existed. Only * the *key* got better (identity rather than raw IP); the numbers are the same. diff --git a/v5/src/styles/admin-mirror.css b/v5/src/styles/admin-mirror.css new file mode 100644 index 0000000..adfbfbf --- /dev/null +++ b/v5/src/styles/admin-mirror.css @@ -0,0 +1,365 @@ +/* + * `/admin/mirror` (spec §3.8, §6). + * + * Imported by `MirrorConnect`, `MirrorMapping`, `MirrorStatus` and + * `MirrorControls`, and nothing else. Every rule reads the `--td-*` tokens + * `.admin-shell` supplies or the global theme tokens, so light and dark follow + * without a palette of its own. Fields, buttons and the status line are the + * shared `.admin-*` rules in globals.css; this file adds the panels, the facts + * list, the mapping table and the problem lines — and makes all of it work at a + * 390px viewport, where the mapping table becomes a stacked list. + */ + +.admin-mirror { + display: flex; + flex-direction: column; + gap: 20px; +} + +.admin-mirror-panel { + display: flex; + flex-direction: column; + gap: 14px; + min-width: 0; + padding: 20px; + border: 1px solid var(--td-line); + background: var(--td-surface); +} + +.admin-mirror-panel h3 { + margin: 0; + font-family: var(--font-display); + font-size: 19px; + font-weight: 500; +} + +.admin-mirror-hint { + margin: 0; + max-width: 68ch; + color: var(--td-muted); + font-size: 13px; + line-height: 1.5; +} + +/* ── The notices: no key, and a token that needs replacing ───────── */ + +.admin-mirror-notice { + padding: 14px 16px; + border-left: 2px solid var(--td-warning); + background: var(--surface-container-low); +} + +.admin-mirror-notice h3 { + margin: 0 0 6px; + font-family: var(--font-display); + font-size: 17px; + font-weight: 500; +} + +.admin-mirror-notice p { + margin: 0; + max-width: 68ch; + line-height: 1.5; +} + +.admin-mirror-steps { + margin: 0; + padding-left: 20px; + max-width: 68ch; + line-height: 1.6; +} + +/* ── Connect ─────────────────────────────────────────────────────── */ + +.admin-mirror-form { + display: flex; + flex-direction: column; + gap: 12px; + max-width: 560px; +} + +.admin-mirror-form input { + width: 100%; + min-width: 0; + box-sizing: border-box; +} + +.admin-mirror-field-hint { + color: var(--td-muted); + font-family: var(--font-body); + font-size: 12px; + letter-spacing: normal; + text-transform: none; +} + +.admin-mirror-actions { + display: flex; + flex-wrap: wrap; + gap: 8px; + align-items: center; +} + +.admin-mirror-line { + margin: 0; + color: var(--td-muted); + font-size: 13px; + line-height: 1.5; +} + +.admin-mirror-line:empty { + display: none; +} + +.admin-mirror-line.is-ok { + color: var(--td-accent); +} + +.admin-mirror-line.is-error { + color: var(--secondary); +} + +.admin-mirror-line.is-warning { + color: var(--td-warning); +} + +/* ── Status ──────────────────────────────────────────────────────── */ + +.admin-mirror-flags { + display: flex; + flex-wrap: wrap; + gap: 8px; + margin: 0; + padding: 0; + list-style: none; +} + +.admin-mirror-flags:empty { + display: none; +} + +.admin-mirror-flag { + padding: 4px 8px; + border: 1px dashed var(--td-line); + color: var(--td-ink-2); + font-family: var(--font-mono); + font-size: 11px; + letter-spacing: 0.06em; +} + +.admin-mirror-flag.is-paused { + border-style: solid; + border-color: var(--td-warning); + color: var(--td-warning); +} + +.admin-mirror-facts { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(160px, 1fr)); + gap: 12px; + margin: 0; +} + +.admin-mirror-facts > div { + display: flex; + flex-direction: column; + gap: 4px; + min-width: 0; +} + +.admin-mirror-facts dt { + color: var(--td-muted); + font-family: var(--font-mono); + font-size: 11px; + letter-spacing: 0.08em; + text-transform: uppercase; +} + +.admin-mirror-facts dd { + margin: 0; + font-size: 14px; + overflow-wrap: anywhere; +} + +.admin-mirror-result.is-ok { + color: var(--td-accent); +} + +.admin-mirror-result.is-partial { + color: var(--td-warning); +} + +.admin-mirror-result.is-failed { + color: var(--secondary); +} + +.admin-mirror-error { + display: flex; + flex-direction: column; + gap: 6px; + padding: 10px 12px; + border-left: 2px solid var(--secondary); + background: var(--surface-container-low); +} + +.admin-mirror-error.is-partial { + border-left-color: var(--td-warning); +} + +.admin-mirror-error p { + margin: 0; + line-height: 1.5; +} + +.admin-mirror-error-label { + color: var(--secondary); + font-family: var(--font-mono); + font-size: 11px; + letter-spacing: 0.08em; + text-transform: uppercase; +} + +.admin-mirror-error.is-partial .admin-mirror-error-label { + color: var(--td-warning); +} + +/* Notion's own words are read by whoever fixes it: keep them, and wrap a long + id rather than widening the panel. */ +.admin-mirror-detail { + font-family: var(--font-mono); + font-size: 12px; + white-space: pre-wrap; + overflow-wrap: anywhere; +} + +/* ── Mapping ─────────────────────────────────────────────────────── */ + +.admin-mirror-mapping { + width: 100%; + border-collapse: collapse; + font-size: 14px; +} + +.admin-mirror-mapping th, +.admin-mirror-mapping td { + padding: 8px 10px; + border-bottom: 1px solid var(--td-line); + text-align: left; + vertical-align: top; +} + +.admin-mirror-mapping thead th { + color: var(--td-muted); + font-family: var(--font-mono); + font-size: 11px; + font-weight: 400; + letter-spacing: 0.08em; + text-transform: uppercase; +} + +.admin-mirror-mapping tbody th { + font-weight: 500; + white-space: nowrap; +} + +.admin-mirror-mapping code { + font-family: var(--font-mono); + font-size: 12px; + overflow-wrap: anywhere; +} + +.admin-mirror-unset { + color: var(--td-muted); + font-style: italic; +} + +.admin-mirror-paste > summary { + padding: 6px 0; + cursor: pointer; + color: var(--td-accent); + font-family: var(--font-mono); + font-size: 12px; + letter-spacing: 0.06em; + text-transform: uppercase; +} + +.admin-mirror-paste[open] > summary { + margin-bottom: 10px; +} + +.admin-mirror-paste-fields { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(240px, 1fr)); + gap: 12px; +} + +.admin-mirror-problems { + margin: 2px 0 0; + padding: 0; + list-style: none; + color: var(--secondary); + font-family: var(--font-body); + font-size: 12px; + letter-spacing: normal; + line-height: 1.4; + text-transform: none; +} + +/* ── Controls ────────────────────────────────────────────────────── */ + +.admin-mirror-confirm { + display: flex; + flex-wrap: wrap; + gap: 8px; + align-items: center; + padding: 10px 12px; + border-left: 2px solid var(--secondary); + background: var(--surface-container-low); +} + +.admin-mirror-confirm p { + flex-basis: 100%; + margin: 0; + max-width: 68ch; + line-height: 1.5; +} + +/* ── A phone ─────────────────────────────────────────────────────── */ + +@media (max-width: 560px) { + .admin-mirror-panel { + padding: 14px; + } + + /* The mapping table becomes a stacked list: the table name over its id. */ + .admin-mirror-mapping thead { + position: absolute; + width: 1px; + height: 1px; + overflow: hidden; + clip: rect(0 0 0 0); + white-space: nowrap; + } + + .admin-mirror-mapping tr { + display: flex; + flex-direction: column; + gap: 2px; + padding: 8px 0; + border-bottom: 1px solid var(--td-line); + } + + .admin-mirror-mapping th, + .admin-mirror-mapping td { + padding: 0; + border-bottom: 0; + } + + .admin-mirror-actions .admin-button { + flex: 1 1 auto; + } +} + +/* The watermark note under "Last synced" sits on its own line beside the time. */ +.admin-mirror-facts dd .admin-mirror-hint { + display: block; +} diff --git a/v5/src/styles/globals.css b/v5/src/styles/globals.css index 367a779..cd51969 100644 --- a/v5/src/styles/globals.css +++ b/v5/src/styles/globals.css @@ -2723,12 +2723,35 @@ p { text-transform: uppercase; } -.td-hero-copy > p { +.td-hero-description { max-width: 620px; - margin: 0; color: var(--td-ink-2); font-size: 16px; line-height: 1.65; + display: grid; + gap: 12px; +} + +.td-hero-description > * { + margin: 0; +} + +.td-hero-description ul, +.td-hero-description ol { + padding-left: 20px; + display: grid; + gap: 2px; +} + +.td-hero-description strong { + color: var(--td-ink); + font-weight: 600; +} + +.td-hero-description a { + color: inherit; + text-decoration: underline; + text-underline-offset: 2px; } .td-chip-row { @@ -3289,7 +3312,7 @@ p { font-size: 26px; } - .td-hero-copy > p { + .td-hero-description { font-size: 15px; overflow-wrap: anywhere; } @@ -3832,8 +3855,7 @@ p { font-weight: 500; } -.admin-lede, -.admin-index-note { +.admin-lede { margin: 0; max-width: 68ch; color: var(--on-surface-muted); diff --git a/v5/src/styles/intake-table.css b/v5/src/styles/intake-table.css index d48cd50..6fc531d 100644 --- a/v5/src/styles/intake-table.css +++ b/v5/src/styles/intake-table.css @@ -76,6 +76,11 @@ width: 32px; } +/* Visible checkbox text exists only for the stacked layout below. */ +.intake-select-text { + display: none; +} + .intake-table input[type="checkbox"] { width: 16px; height: 16px; @@ -336,6 +341,29 @@ display: none; } + /* Stacked, a checkbox loses its column header, so it says what it does. */ + .intake-table td.intake-cell-select, + .intake-table thead th.intake-col-select { + display: flex; + align-items: center; + gap: 8px; + } + + .intake-select-text { + display: inline; + font-family: var(--font-mono); + font-size: 11px; + letter-spacing: 0.04em; + text-transform: uppercase; + color: var(--on-surface); + } + + .intake-row.is-unresolved .intake-select-text { + color: var(--on-surface-muted); + text-transform: none; + letter-spacing: 0; + } + .intake-row.is-unresolved td { background: transparent; } diff --git a/v5/src/workflows/archive-manuals.test.ts b/v5/src/workflows/archive-manuals.test.ts new file mode 100644 index 0000000..9851f7d --- /dev/null +++ b/v5/src/workflows/archive-manuals.test.ts @@ -0,0 +1,31 @@ +// @vitest-environment node + +/** + * `archiveManuals` as a plain function: without the workflow compiler the + * directives are strings, so the steps are mocked and the orchestration is + * under test — one step per resource, in order, each ending on its own. + */ + +const steps = vi.hoisted(() => ({ archiveManualStep: vi.fn(), finishManualArchive: vi.fn() })); +vi.mock("../lib/manuals/steps", () => steps); + +import { archiveManuals } from "./archive-manuals"; + +beforeEach(() => { + steps.archiveManualStep.mockReset(); + steps.finishManualArchive.mockReset().mockResolvedValue(undefined); +}); + +describe("archiveManuals", () => { + it("archives each resource in order and counts every outcome, a thrown step as failed", async () => { + steps.archiveManualStep + .mockResolvedValueOnce({ status: "archived", reason: "archived" }) + .mockRejectedValueOnce(new Error("retries exhausted")) + .mockResolvedValueOnce({ status: "skipped", reason: "already_archived" }) + .mockResolvedValueOnce({ status: "failed", reason: "not_pdf", transient: false }); + + expect(await archiveManuals(["a", "b", "c", "d"])).toEqual({ archived: 1, skipped: 1, failed: 2 }); + expect(steps.archiveManualStep.mock.calls.map(([id]) => id)).toEqual(["a", "b", "c", "d"]); + expect(steps.finishManualArchive).toHaveBeenCalledWith({ archived: 1, skipped: 1, failed: 2 }); + }); +}); diff --git a/v5/src/workflows/archive-manuals.ts b/v5/src/workflows/archive-manuals.ts new file mode 100644 index 0000000..a046877 --- /dev/null +++ b/v5/src/workflows/archive-manuals.ts @@ -0,0 +1,32 @@ +import { archiveManualStep, finishManualArchive } from "../lib/manuals/steps.ts"; + +/** + * `archiveManuals` — copy each resource's manual PDF into Blob + * (`src/lib/manuals/archive.ts`), one step per resource. + * + * Started, through `src/lib/manuals/start.ts`, after approving a tool (the + * resources it created), after `create_tool` over MCP, after the tool editor + * adds a resource or changes its link, and by the daily cron's backfill. + * + * **Sequential, in the order given.** The body is replayed from the run's + * event log after every step, and a replay must issue the same step calls in + * the same order; a handful of PDFs does not need a pool. Each resource ends + * independently: a step that is still failing after its retries counts as + * failed and the rest are archived. + */ +export async function archiveManuals( + resourceIds: string[] +): Promise<{ archived: number; skipped: number; failed: number }> { + "use workflow"; + const counts = { archived: 0, skipped: 0, failed: 0 }; + for (const id of resourceIds) { + try { + const result = await archiveManualStep(id); + counts[result.status] += 1; + } catch { + counts.failed += 1; + } + } + await finishManualArchive(counts); + return counts; +} diff --git a/v5/src/workflows/mirror-push.test.ts b/v5/src/workflows/mirror-push.test.ts new file mode 100644 index 0000000..b58eb48 --- /dev/null +++ b/v5/src/workflows/mirror-push.test.ts @@ -0,0 +1,201 @@ +// @vitest-environment node + +/** + * `mirrorPush` and `mirrorPushAfterChange` as plain functions (spec §10, the + * 2026-09-22 amendment). Without the workflow compiler `"use workflow"` and + * `"use step"` are only strings, so the steps module and `sleep` are mocked + * and the orchestration is what is under test: rounds while a push runs out + * of budget, the cap, the sleep before a coalesced push, and the same calls + * in the same order every time — which is what a replay needs. + */ + +const log = vi.hoisted(() => [] as string[]); + +const steps = vi.hoisted(() => ({ + pushMirrorRound: vi.fn(), + takeCoalescedMirrors: vi.fn(), + finishMirrorPush: vi.fn(), +})); + +const wf = vi.hoisted(() => ({ sleep: vi.fn() })); + +vi.mock("../lib/mirror/steps", () => steps); +vi.mock("workflow", () => ({ sleep: wf.sleep })); + +import { MIRROR_BUSY_PAUSE, MIRROR_BUSY_RETRIES, MIRROR_COALESCE_DELAY, MIRROR_MAX_ROUNDS } from "../lib/mirror/limits"; +import type { MirrorPushOutcome } from "../lib/mirror/push"; +import { mirrorPush, mirrorPushAfterChange } from "./mirror-push"; + +const ERROR = { code: "rows_failed" as const, entities: ["tools" as const], failed: 1, detail: null }; + +const INCOMPLETE: MirrorPushOutcome = { state: "incomplete", pushed: 40, archived: 1 }; +const OK: MirrorPushOutcome = { state: "ok", pushed: 3, archived: 0 }; + +/** Each call to `pushMirrorRound(id)` answers the next outcome queued for that id. */ +function rounds(plan: Record) { + const queues = new Map(Object.entries(plan).map(([id, outcomes]) => [id, [...outcomes]])); + steps.pushMirrorRound.mockImplementation(async (id: string) => { + log.push(`push:${id}`); + const next = queues.get(id)?.shift(); + if (!next) throw new Error(`no outcome planned for ${id}`); + return next; + }); +} + +beforeEach(() => { + log.length = 0; + for (const fn of Object.values(steps)) fn.mockReset(); + wf.sleep.mockReset().mockImplementation(async (duration: string) => { + log.push(`sleep:${duration}`); + }); + steps.finishMirrorPush.mockImplementation(async () => { + log.push("finish"); + }); +}); + +describe("mirrorPush", () => { + it("runs another round while a push is incomplete, pausing between rounds, and stops at ok", async () => { + rounds({ m1: [INCOMPLETE, INCOMPLETE, OK] }); + + expect(await mirrorPush("m1")).toEqual({ + mirrorId: "m1", + rounds: 3, + state: "ok", + pushed: 83, + archived: 2, + failed: 0, + }); + expect(log).toEqual(["push:m1", "sleep:5s", "push:m1", "sleep:5s", "push:m1", "finish"]); + expect(steps.finishMirrorPush).toHaveBeenCalledWith("run", [expect.objectContaining({ mirrorId: "m1", state: "ok" })]); + }); + + it.each<[string, MirrorPushOutcome]>([ + ["ok", OK], + ["partial", { state: "partial", pushed: 5, archived: 0, failed: 1, error: ERROR }], + ["failed", { state: "failed", error: { ...ERROR, code: "unauthorized" }, paused: true }], + ["skipped", { state: "skipped", reason: "paused" }], + ])("stops after one round that ends %s", async (state, outcome) => { + rounds({ m1: [outcome, OK] }); + + const summary = await mirrorPush("m1"); + + expect(summary.rounds).toBe(1); + expect(summary.state).toBe(state); + expect(steps.pushMirrorRound).toHaveBeenCalledTimes(1); + expect(wf.sleep).not.toHaveBeenCalled(); + }); + + it("waits out another push that holds the mirror, then pushes, without spending a round", async () => { + const BUSY: MirrorPushOutcome = { state: "skipped", reason: "running" }; + rounds({ m1: [BUSY, BUSY, INCOMPLETE, OK] }); + + expect(await mirrorPush("m1")).toMatchObject({ rounds: 4, state: "ok", pushed: 43 }); + expect(log).toEqual([ + "push:m1", + `sleep:${MIRROR_BUSY_PAUSE}`, + "push:m1", + `sleep:${MIRROR_BUSY_PAUSE}`, + "push:m1", + "sleep:5s", + "push:m1", + "finish", + ]); + }); + + it(`gives up as skipped after ${MIRROR_BUSY_RETRIES} waits for a push that never finishes`, async () => { + rounds({ m1: Array.from({ length: MIRROR_BUSY_RETRIES + 3 }, () => ({ state: "skipped", reason: "running" }) as const) }); + + expect(await mirrorPush("m1")).toMatchObject({ rounds: MIRROR_BUSY_RETRIES + 1, state: "skipped" }); + expect(wf.sleep).toHaveBeenCalledTimes(MIRROR_BUSY_RETRIES); + }); + + it("counts the failed rows of a partial round after incomplete ones", async () => { + rounds({ m1: [INCOMPLETE, { state: "partial", pushed: 2, archived: 0, failed: 3, error: ERROR }] }); + expect(await mirrorPush("m1")).toMatchObject({ rounds: 2, state: "partial", pushed: 42, failed: 3 }); + }); + + it(`stops after ${MIRROR_MAX_ROUNDS} rounds even when every one is incomplete`, async () => { + rounds({ m1: Array.from({ length: MIRROR_MAX_ROUNDS + 3 }, () => INCOMPLETE) }); + + const summary = await mirrorPush("m1"); + + expect(summary).toMatchObject({ rounds: MIRROR_MAX_ROUNDS, state: "incomplete" }); + expect(steps.pushMirrorRound).toHaveBeenCalledTimes(MIRROR_MAX_ROUNDS); + expect(wf.sleep).toHaveBeenCalledTimes(MIRROR_MAX_ROUNDS - 1); + }); + + it("ends as error, and still logs, when a round throws after its retries", async () => { + steps.pushMirrorRound.mockRejectedValue(new Error("Mirror push: the database could not be reached.")); + + expect(await mirrorPush("m1")).toMatchObject({ rounds: 1, state: "error" }); + expect(steps.finishMirrorPush).toHaveBeenCalledWith("run", [expect.objectContaining({ state: "error" })]); + }); +}); + +describe("mirrorPushAfterChange", () => { + it("sleeps first, then takes the waiting mirrors and pushes each in order", async () => { + steps.takeCoalescedMirrors.mockImplementation(async () => { + log.push("take"); + return ["m1", "m2", "m3"]; + }); + rounds({ m1: [OK], m2: [INCOMPLETE, OK], m3: [{ state: "skipped", reason: "paused" }] }); + + const result = await mirrorPushAfterChange(); + + expect(log).toEqual([ + `sleep:${MIRROR_COALESCE_DELAY}`, + "take", + "push:m1", + "push:m2", + "sleep:5s", + "push:m2", + "push:m3", + "finish", + ]); + expect(result.mirrors.map((m) => [m.mirrorId, m.state, m.rounds])).toEqual([ + ["m1", "ok", 1], + ["m2", "ok", 2], + ["m3", "skipped", 1], + ]); + expect(steps.finishMirrorPush).toHaveBeenCalledWith("change", result.mirrors); + }); + + it("honours an explicit delay", async () => { + steps.takeCoalescedMirrors.mockResolvedValue([]); + await mirrorPushAfterChange("30s"); + expect(wf.sleep).toHaveBeenCalledWith("30s"); + }); + + it("pushes the next mirror when one throws", async () => { + steps.takeCoalescedMirrors.mockResolvedValue(["m1", "m2"]); + steps.pushMirrorRound.mockImplementation(async (id: string) => { + if (id === "m1") throw new Error("boom"); + return OK; + }); + + const result = await mirrorPushAfterChange(); + + expect(result.mirrors.map((m) => m.state)).toEqual(["error", "ok"]); + }); + + it("pushes nothing when nothing is waiting", async () => { + steps.takeCoalescedMirrors.mockResolvedValue([]); + expect(await mirrorPushAfterChange()).toEqual({ mirrors: [] }); + expect(steps.pushMirrorRound).not.toHaveBeenCalled(); + expect(steps.finishMirrorPush).toHaveBeenCalledWith("change", []); + }); + + it("issues the same calls in the same order every run", async () => { + const plan = { m1: [INCOMPLETE, OK], m2: [OK] }; + steps.takeCoalescedMirrors.mockResolvedValue(["m1", "m2"]); + rounds(plan); + await mirrorPushAfterChange(); + const first = [...log]; + + log.length = 0; + rounds(plan); + await mirrorPushAfterChange(); + + expect(log).toEqual(first); + }); +}); diff --git a/v5/src/workflows/mirror-push.ts b/v5/src/workflows/mirror-push.ts new file mode 100644 index 0000000..3284b2b --- /dev/null +++ b/v5/src/workflows/mirror-push.ts @@ -0,0 +1,124 @@ +import { sleep } from "workflow"; +import { MIRROR_BUSY_PAUSE, MIRROR_BUSY_RETRIES, MIRROR_COALESCE_DELAY, MIRROR_MAX_ROUNDS } from "../lib/mirror/limits.ts"; +import type { MirrorPushOutcome } from "../lib/mirror/push.ts"; +import { + finishMirrorPush, + pushMirrorRound, + takeCoalescedMirrors, + type MirrorPushSummary, +} from "../lib/mirror/steps.ts"; + +/** + * The Notion mirror's two workflows (spec §3.8 "Push" and "Triggers"). + * + * - {@link mirrorPush} — push one mirror now. Started by **Sync now** + * (`syncMirrorNow`) and by the daily cron's backstop (`runMirrorBackstop`), + * both through `src/lib/mirror/start.ts`. + * - {@link mirrorPushAfterChange} — a change in the app (approving a tool, + * publishing, saving an edit) called `requestMirrorPush()`; sleep two + * minutes so a burst of edits becomes one push, then push every active + * mirror, one after another. + * + * **Rounds.** One push is bounded at 45 seconds (§3.8, `MIRROR_PUSH_BUDGET_MS`) + * so each step stays far inside the Hobby plan's 300-second function ceiling. + * A first sync of a real inventory is bigger than that, so a round that ran + * out of budget with nothing failed answers `incomplete`, and the workflow + * pauses five seconds and runs another — up to {@link MIRROR_MAX_ROUNDS}. + * Anything left after that waits for the next trigger or the nightly + * backstop. Every other state — `ok`, `partial`, `failed`, `skipped` — ends + * that mirror's rounds: a failure is retried by the next trigger, not in a + * loop here (§5.8). + * + * **Busy.** One exception: a round skipped as `running` — another push holds + * the mirror — waits {@link MIRROR_BUSY_PAUSE} and claims again, up to + * {@link MIRROR_BUSY_RETRIES} times, without spending a round. The other push + * may have read its tables before the change that started this one, so giving + * up would leave that change for the nightly backstop. + * + * **Deterministic for replay.** The body is replayed from the run's event log + * after every step, and a replay must issue the same step calls in the same + * order. So the mirrors are pushed sequentially in the order the step + * returned them, nothing reads the clock or draws a random number, and there + * is no worker pool. The steps do everything that touches the database or + * Notion (`src/lib/mirror/steps.ts`). + */ + +/** The pause between rounds of one mirror's push. */ +const ROUND_PAUSE = "5s"; + +/** A duration `sleep` accepts, in the form `limits.ts` spells them. */ +type SleepDuration = `${number}${"s" | "m" | "h"}`; + +/** Push one mirror, in as many rounds as it takes (at most {@link MIRROR_MAX_ROUNDS}). */ +export async function mirrorPush(mirrorId: string): Promise { + "use workflow"; + const summary = await pushInRounds(mirrorId); + await finishMirrorPush("run", [summary]); + return summary; +} + +/** + * Wait for a burst of edits to finish, then push every mirror that asked for + * it. The waiting claims are taken *after* the sleep, so an edit made during + * it rides along, and one made while the pushes run claims — and starts — the + * next run. + */ +export async function mirrorPushAfterChange( + delay: SleepDuration = MIRROR_COALESCE_DELAY +): Promise<{ mirrors: MirrorPushSummary[] }> { + "use workflow"; + await sleep(delay); + const ids = await takeCoalescedMirrors(); + const mirrors: MirrorPushSummary[] = []; + for (const id of ids) { + mirrors.push(await pushInRounds(id)); + } + await finishMirrorPush("change", mirrors); + return { mirrors }; +} + +/** + * One mirror's rounds. A plain function in workflow scope, not a step. + * + * A round that throws has already been retried by the SDK and has released + * its claim (`pushMirror`'s `finally`); it ends this mirror as `error` and + * the next mirror still gets its push — one mirror's database trouble must + * not strand another admin's. + */ +async function pushInRounds(mirrorId: string): Promise { + const summary: MirrorPushSummary = { mirrorId, rounds: 0, state: "incomplete", pushed: 0, archived: 0, failed: 0 }; + /** Rounds that did work, which {@link MIRROR_MAX_ROUNDS} bounds; `summary.rounds` counts every call. */ + let worked = 0; + let busy = 0; + let pause: SleepDuration | null = null; + + while (worked < MIRROR_MAX_ROUNDS) { + if (pause) await sleep(pause); + summary.rounds += 1; + + let outcome: MirrorPushOutcome; + try { + outcome = await pushMirrorRound(mirrorId); + } catch { + summary.state = "error"; + break; + } + + summary.state = outcome.state; + if (outcome.state === "skipped" && outcome.reason === "running" && busy < MIRROR_BUSY_RETRIES) { + busy += 1; + pause = MIRROR_BUSY_PAUSE; + continue; + } + worked += 1; + pause = ROUND_PAUSE; + if (outcome.state === "ok" || outcome.state === "incomplete" || outcome.state === "partial") { + summary.pushed += outcome.pushed; + summary.archived += outcome.archived; + } + if (outcome.state === "partial") summary.failed += outcome.failed; + if (outcome.state !== "incomplete") break; + } + + return summary; +} diff --git a/v5/src/workflows/mirror-push.workflow.test.ts b/v5/src/workflows/mirror-push.workflow.test.ts new file mode 100644 index 0000000..587e897 --- /dev/null +++ b/v5/src/workflows/mirror-push.workflow.test.ts @@ -0,0 +1,111 @@ +import { eq, and } from "drizzle-orm"; +import { start } from "workflow/api"; +import { createNotionFake } from "../../test/fakes/notion-fake"; +import { useNotionFake as installNotionFake } from "../../test/msw/notion-mirror"; +import { server } from "../../test/msw/server"; +import { getMirror, saveMirrorConnection } from "../lib/data/mirrors"; +import { getDb, resetDbForTests } from "../lib/db/client"; +import { DEMO_ACCOUNTS } from "../lib/db/demo-seed"; +import { mirrorPages, notionMirrors, tools } from "../lib/db/schema/index"; +import { ensureMirrorDatabases } from "../lib/mirror/databases"; +import { encryptMirrorToken } from "../lib/mirror/token-crypto"; +import { mirrorPush } from "./mirror-push"; + +/** + * The one in-process `@workflow/vitest` run of `mirrorPush` (spec §10, the + * mirror's integration tier): the real workflow runtime, the real step + * bundle, the seeded PGlite database, and the fake Notion answering on + * `api.notion.com` through MSW — which, unlike `vi.mock()`, reaches step code + * (the 2026-09-22 amendment). No network and no real credential: the token + * and `AUTH_SECRET` are made up here. + * + * The step bundle has its own copy of `db/client.ts`, but that module keeps + * its handle on `globalThis`, so calling `getDb()` here first means the steps + * find this same PGlite database. + */ + +const AUTH_SECRET = "mirror-workflow-test-secret-0123456789"; +const TOKEN = `ntn_${"W0rkfl0wT3st".repeat(4)}`; + +beforeAll(async () => { + vi.stubEnv("DATABASE_URL", ""); + await getDb(); +}); + +afterAll(() => { + resetDbForTests(); +}); + +describe("mirrorPush (in process)", () => { + it("pushes the demo inventory into the fake Notion and records ok", { timeout: 120_000 }, async () => { + vi.stubEnv("DATABASE_URL", ""); + vi.stubEnv("AUTH_SECRET", AUTH_SECRET); + + const printed: unknown[][] = []; + for (const method of ["log", "info", "warn", "error", "debug"] as const) { + vi.spyOn(console, method).mockImplementation((...args: unknown[]) => { + printed.push(args); + }); + } + + const fake = createNotionFake({ token: TOKEN }); + const parentPageId = fake.addPage({ title: "MakerLab mirror" }); + installNotionFake(server, fake); + + const db = await getDb(); + await db.delete(notionMirrors); + const { mirror } = await saveMirrorConnection({ + ownerUserId: DEMO_ACCOUNTS.admin.id, + tokenCiphertext: encryptMirrorToken(TOKEN, AUTH_SECRET), + parentPageId, + parentPageTitle: "MakerLab mirror", + }); + + const ensured = await ensureMirrorDatabases(mirror.id); + expect(ensured.ok).toBe(true); + if (!ensured.ok) throw new Error("unreachable"); + const toolsDatabase = ensured.mapping.tools; + expect(toolsDatabase).toBeTruthy(); + + const run = await start(mirrorPush, [mirror.id]); + const summary = await run.returnValue; + + expect(summary).toMatchObject({ mirrorId: mirror.id, state: "ok" }); + expect(summary.pushed).toBeGreaterThan(0); + + const after = await getMirror(mirror.id); + expect(after?.lastStatus).toBe("ok"); + expect(after?.lastSyncedAt).not.toBeNull(); + expect(after?.lastError).toBeNull(); + expect(after?.runningSince).toBeNull(); + + // Every tool — published or not — has a page in the tools database, and + // a mirror_pages row pointing at it. + const demoTools = await db.select({ id: tools.id }).from(tools); + expect(demoTools.length).toBeGreaterThan(0); + expect(fake.pagesIn(toolsDatabase!)).toHaveLength(demoTools.length); + const recorded = await db + .select({ entityId: mirrorPages.entityId, notionPageId: mirrorPages.notionPageId }) + .from(mirrorPages) + .where(and(eq(mirrorPages.mirrorId, mirror.id), eq(mirrorPages.entity, "tools"))); + expect(recorded.map((row) => row.entityId).sort()).toEqual(demoTools.map((row) => row.id).sort()); + for (const row of recorded) expect(fake.pages.has(row.notionPageId)).toBe(true); + + // Never the token, anywhere a log line could carry it (§10 "cases that + // would embarrass us"), nor the secret it is encrypted under. + const output = printed.map((args) => args.map(printable).join(" ")).join("\n"); + expect(output).not.toContain(TOKEN); + expect(output).not.toContain(AUTH_SECRET); + }); +}); + +/** One console argument as text, whatever it is. */ +function printable(arg: unknown): string { + if (typeof arg === "string") return arg; + if (arg instanceof Error) return `${arg.message} ${arg.stack ?? ""} ${printable(arg.cause)}`; + try { + return JSON.stringify(arg) ?? String(arg); + } catch { + return String(arg); + } +} diff --git a/v5/test/fakes/notion-fake.test.ts b/v5/test/fakes/notion-fake.test.ts new file mode 100644 index 0000000..47d27ba --- /dev/null +++ b/v5/test/fakes/notion-fake.test.ts @@ -0,0 +1,178 @@ +// @vitest-environment node +import { NotionMirrorError, createNotionClient, pageTitle } from "../../src/lib/mirror/notion-client"; +import { server } from "../msw/server"; +// Aliased: the name starts with `use`, which eslint's rules-of-hooks reads as a React hook. +import { useNotionFake as installNotionFake } from "../msw/notion-mirror"; +import { createNotionFake } from "./notion-fake"; + +/** + * A self-test of the fake Notion, driven through the real mirror client over + * MSW — so what the parts build against is known to speak the client's + * dialect before anybody writes a push against it. + */ +const TOKEN = "ntn_FAKEtoken0123456789abcdef"; +const PARENT = "0f5e4a3c-1111-2222-3333-44445555aaaa"; + +function setup() { + const fake = createNotionFake({ token: TOKEN, pages: [{ id: PARENT, title: "MakerLab Tools — mirror" }] }); + installNotionFake(server, fake); + const client = createNotionClient({ token: TOKEN, requestsPerSecond: 0 }); + return { fake, client }; +} + +async function codeOf(promise: Promise): Promise { + try { + await promise; + } catch (error) { + return (error as NotionMirrorError).code; + } + return "resolved"; +} + +describe("createNotionFake", () => { + it("reads a seeded page's title, dashed or not", async () => { + const { client } = setup(); + expect(pageTitle(await client.getPage(PARENT))).toBe("MakerLab Tools — mirror"); + expect((await client.getPage(PARENT.replace(/-/g, ""))).id).toBe(PARENT); + expect(await codeOf(client.getPage(crypto.randomUUID()))).toBe("page_not_found"); + }); + + it("refuses the wrong token with Notion's 401, and never logs a header", async () => { + const { fake } = setup(); + const wrong = createNotionClient({ token: "ntn_wrong", requestsPerSecond: 0 }); + const error = await wrong.getPage(PARENT).catch((e: NotionMirrorError) => e); + expect(error).toBeInstanceOf(NotionMirrorError); + expect((error as NotionMirrorError).code).toBe("unauthorized"); + expect((error as NotionMirrorError).message).toBe("Notion 401 unauthorized: API token is invalid."); + expect(JSON.stringify(fake.requests)).not.toContain("ntn_"); + }); + + it("creates a database with its properties echoed, then merges a PATCH", async () => { + const { fake, client } = setup(); + const categories = await client.createDatabase({ + parent: { type: "page_id", page_id: PARENT }, + title: [{ type: "text", text: { content: "Categories" } }], + properties: { Name: { title: {} } }, + }); + const tools = await client.createDatabase({ + parent: { type: "page_id", page_id: PARENT }, + title: [{ type: "text", text: { content: "Tools" } }], + description: [{ type: "text", text: { content: "Mirrored from MakerLab Tools. Edits here are overwritten." } }], + properties: { + Name: { title: {} }, + Published: { checkbox: {} }, + Category: { relation: { database_id: categories.id, single_property: {} } }, + }, + }); + expect(tools.properties.Published).toMatchObject({ name: "Published", type: "checkbox" }); + expect(tools.properties.Category.relation?.database_id).toBe(categories.id); + expect(tools.properties.Name.id).toEqual(expect.any(String)); + + const patched = await client.updateDatabase(tools.id, { properties: { Brand: { rich_text: {} }, Published: null } }); + expect(Object.keys(patched.properties).sort()).toEqual(["Brand", "Category", "Name"]); + expect((await client.getDatabase(tools.id)).properties.Brand.type).toBe("rich_text"); + expect(fake.databases.size).toBe(2); + }); + + it("refuses a database under an unknown page, and one without a title property", async () => { + const { client } = setup(); + expect( + await codeOf(client.createDatabase({ parent: { page_id: crypto.randomUUID() }, properties: { Name: { title: {} } } })) + ).toBe("page_not_found"); + expect(await codeOf(client.createDatabase({ parent: { page_id: PARENT }, properties: { Notes: { rich_text: {} } } }))).toBe( + "validation" + ); + }); + + it("creates, updates and archives pages in a database, checking values against the schema", async () => { + const { fake, client } = setup(); + const db = await client.createDatabase({ + parent: { page_id: PARENT }, + properties: { Name: { title: {} }, Published: { checkbox: {} } }, + }); + + const created = await client.createPage({ + parent: { database_id: db.id }, + properties: { Name: { title: [{ text: { content: "Form 4" } }] } }, + }); + expect(fake.pagesIn(db.id)).toHaveLength(1); + const stored = await client.getPage(created.id); + expect(pageTitle(stored)).toBe("Form 4"); + expect(stored.properties.Published).toMatchObject({ type: "checkbox", checkbox: false }); + + await client.updatePage(created.id, { properties: { Published: { checkbox: true } } }); + expect((await client.getPage(created.id)).properties.Published.checkbox).toBe(true); + + await client.updatePage(created.id, { archived: true }); + expect((await client.getPage(created.id)).archived).toBe(true); + // Notion refuses to edit an archived page until it is unarchived. + expect(await codeOf(client.updatePage(created.id, { properties: { Published: { checkbox: false } } }))).toBe("validation"); + await client.updatePage(created.id, { archived: false, properties: { Published: { checkbox: false } } }); + + expect(await codeOf(client.createPage({ parent: { database_id: db.id }, properties: { Nope: { checkbox: true } } }))).toBe( + "validation" + ); + expect(await codeOf(client.createPage({ parent: { database_id: db.id }, properties: { Published: { rich_text: [] } } }))).toBe( + "validation" + ); + expect(await codeOf(client.createPage({ parent: { database_id: crypto.randomUUID() }, properties: {} }))).toBe( + "database_not_found" + ); + expect(await codeOf(client.updatePage(crypto.randomUUID(), { archived: true }))).toBe("page_not_found"); + }); + + it("refuses a page into an archived or deleted database", async () => { + const { fake, client } = setup(); + const db = await client.createDatabase({ parent: { page_id: PARENT }, properties: { Name: { title: {} } } }); + await client.updateDatabase(db.id, { archived: true }); + expect(await codeOf(client.createPage({ parent: { database_id: db.id }, properties: {} }))).toBe("database_not_found"); + + fake.databases.delete(db.id); + expect(await codeOf(client.getDatabase(db.id))).toBe("database_not_found"); + }); + + it("injects failures for the next N matching requests, with Retry-After", async () => { + const { fake } = setup(); + const sleeps: number[] = []; + const client = createNotionClient({ + token: TOKEN, + requestsPerSecond: 0, + // A clock that stands still, so each wait is exactly Retry-After. + now: () => 0, + sleep: async (ms) => { + sleeps.push(ms); + }, + }); + fake.failNext({ method: "GET", path: /^\/pages\// }, { status: 429, retryAfter: 2 }, 2); + + expect((await client.getPage(PARENT)).id).toBe(PARENT); + expect(sleeps).toEqual([2000, 2000]); + expect(fake.requests.filter((request) => request.method === "GET")).toHaveLength(3); + + fake.failNext({ path: /^\/databases$/ }, { status: 503 }); + expect(await codeOf(client.createDatabase({ parent: { page_id: PARENT }, properties: { Name: { title: {} } } }))).toBe( + "unavailable" + ); + expect(fake.requests.at(-1)).toMatchObject({ method: "POST", path: "/databases", body: expect.any(Object) }); + }); + + it("resets to the seeded pages", async () => { + const { fake, client } = setup(); + await client.createDatabase({ parent: { page_id: PARENT }, properties: { Name: { title: {} } } }); + const extra = fake.addPage({ title: "Another page" }); + fake.reset(); + expect(fake.databases.size).toBe(0); + expect(fake.requests).toHaveLength(0); + expect(fake.pages.has(PARENT)).toBe(true); + expect(fake.pages.has(extra)).toBe(false); + }); + + it("serves a custom base URL, as the E2E stub is reached", async () => { + const fake = createNotionFake({ token: TOKEN, pages: [{ id: PARENT, title: "Stub" }] }); + installNotionFake(server, fake, "http://127.0.0.1:3102/v1"); + const client = createNotionClient({ token: TOKEN, baseUrl: "http://127.0.0.1:3102/v1", requestsPerSecond: 0 }); + expect(pageTitle(await client.getPage(PARENT))).toBe("Stub"); + expect(fake.handle("GET", "/v1/pages/" + PARENT, { Authorization: `Bearer ${TOKEN}`, "Notion-Version": "2022-06-28" }, undefined).status).toBe(200); + expect(fake.handle("GET", "/pages/" + PARENT, { Authorization: `Bearer ${TOKEN}` }, undefined).status).toBe(400); + }); +}); diff --git a/v5/test/fakes/notion-fake.ts b/v5/test/fakes/notion-fake.ts new file mode 100644 index 0000000..77858ec --- /dev/null +++ b/v5/test/fakes/notion-fake.ts @@ -0,0 +1,555 @@ +/** + * A stateful, in-memory Notion — the 2022-06-28 endpoints the mirror uses + * (spec §10 "Mirror against mocked Notion", E2E scenario 8). + * + * One model, two transports: `test/msw/notion-mirror.ts` installs it as MSW + * handlers for Vitest (including the workflow project, where MSW is the only + * thing that reaches step code), and the E2E stub server calls `handle()` from + * a plain `node:http` server reached through `NOTION_API_BASE_URL`. So this + * file is pure: no imports, no enums, no parameter properties — Node's + * `--experimental-strip-types` loads it as it is. + * + * What it models, and deliberately no more: + * + * - `GET /pages/:id` — a seeded workspace page (the page an admin shares with + * the integration), or a page created in a database. + * - `POST /databases`, `GET /databases/:id`, `PATCH /databases/:id` (merges + * properties; `null` removes one). + * - `POST /pages` into a database, `PATCH /pages/:id` (properties, archived). + * - The bearer token (401 `unauthorized` otherwise) and the `Notion-Version` + * header (400 `missing_version`). + * - 404 `object_not_found` for an unknown id; a page create into an unknown or + * archived database is refused the same way. + * - Property values are checked against the database's schema: an unknown + * property name or a value of the wrong type is a 400 `validation_error`, so + * a push that drifts from the schema it created fails here as it would there. + * - `failNext` injects a status (and `Retry-After`, and a code) for the next N + * matching requests. + * + * `requests` logs method, path, body and a timestamp — **never a header**, so + * the token cannot reach a test's output through it. + */ + +export interface NotionFakeResponse { + status: number; + headers: Record; + body: unknown; +} + +export interface NotionFakeRequest { + method: string; + path: string; + body: unknown; + at: number; +} + +export interface NotionFakeFailure { + status: number; + /** Seconds, sent as `Retry-After`. */ + retryAfter?: number; + /** Notion's error code; derived from the status when omitted. */ + code?: string; + message?: string; +} + +export interface NotionFakeMatch { + method?: string; + path?: RegExp; +} + +export interface FakeProperty { + id: string; + name: string; + type: string; + [key: string]: unknown; +} + +export interface FakeDatabase { + object: "database"; + id: string; + parent: { type: "page_id"; page_id: string }; + title: RichText[]; + description: RichText[]; + properties: Record; + archived: boolean; + in_trash: boolean; + created_time: string; + last_edited_time: string; +} + +export interface FakePage { + object: "page"; + id: string; + parent: { type: "workspace"; workspace: true } | { type: "database_id"; database_id: string }; + properties: Record; + archived: boolean; + in_trash: boolean; + created_time: string; + last_edited_time: string; +} + +interface RichText { + type?: string; + text?: { content?: string }; + plain_text?: string; + [key: string]: unknown; +} + +export interface NotionFake { + handle(method: string, path: string, headers: Record | Headers, body: unknown): NotionFakeResponse; + databases: Map; + pages: Map; + requests: NotionFakeRequest[]; + failNext(match: NotionFakeMatch, response: NotionFakeFailure, times?: number): void; + /** Seed a workspace page (one an integration has been shared with). Returns its id. */ + addPage(page: { id?: string; title: string }): string; + /** The pages created in one database, in creation order. */ + pagesIn(databaseId: string): FakePage[]; + /** Forget every database, created page, request and injected failure; re-seed the initial pages. */ + reset(): void; +} + +export interface NotionFakeOptions { + token: string; + pages?: { id: string; title: string }[]; +} + +/** Property types whose value on a page is keyed by the type name. */ +const PROPERTY_TYPES = new Set([ + "title", + "rich_text", + "number", + "select", + "multi_select", + "status", + "date", + "people", + "files", + "checkbox", + "url", + "email", + "phone_number", + "relation", + "formula", + "rollup", + "created_time", + "created_by", + "last_edited_time", + "last_edited_by", + "unique_id", +]); + +const JSON_HEADERS = { "Content-Type": "application/json" }; + +export function createNotionFake(options: NotionFakeOptions): NotionFake { + const databases = new Map(); + const pages = new Map(); + const requests: NotionFakeRequest[] = []; + let failures: { match: NotionFakeMatch; response: NotionFakeFailure; remaining: number }[] = []; + + function addPage(page: { id?: string; title: string }): string { + const id = normalizeId(page.id ?? "") ?? randomId(); + const at = new Date().toISOString(); + pages.set(id, { + object: "page", + id, + parent: { type: "workspace", workspace: true }, + properties: { title: { id: "title", type: "title", title: [richText(page.title)] } }, + archived: false, + in_trash: false, + created_time: at, + last_edited_time: at, + }); + return id; + } + + function seed(): void { + for (const page of options.pages ?? []) addPage(page); + } + + function reset(): void { + databases.clear(); + pages.clear(); + requests.length = 0; + failures = []; + seed(); + } + + function failNext(match: NotionFakeMatch, response: NotionFakeFailure, times = 1): void { + failures.push({ match, response, remaining: times }); + } + + function takeFailure(method: string, path: string): NotionFakeFailure | null { + for (const failure of failures) { + if (failure.remaining <= 0) continue; + if (failure.match.method && failure.match.method.toUpperCase() !== method) continue; + if (failure.match.path && !failure.match.path.test(path)) continue; + failure.remaining -= 1; + return failure.response; + } + return null; + } + + function pagesIn(databaseId: string): FakePage[] { + const id = normalizeId(databaseId); + return [...pages.values()].filter((page) => page.parent.type === "database_id" && page.parent.database_id === id); + } + + function handle( + rawMethod: string, + rawPath: string, + headers: Record | Headers, + rawBody: unknown + ): NotionFakeResponse { + const method = rawMethod.toUpperCase(); + const path = normalizePath(rawPath); + const parsed = parseBody(rawBody); + requests.push({ method, path, body: parsed.ok ? parsed.value : rawBody, at: Date.now() }); + + const injected = takeFailure(method, path); + if (injected) { + const extra: Record = {}; + if (injected.retryAfter !== undefined) extra["Retry-After"] = String(injected.retryAfter); + return error( + injected.status, + injected.code ?? codeForStatus(injected.status), + injected.message ?? `Injected ${injected.status}.`, + extra + ); + } + + if (header(headers, "authorization") !== `Bearer ${options.token}`) { + return error(401, "unauthorized", "API token is invalid."); + } + if (!header(headers, "notion-version")) { + return error(400, "missing_version", "Notion-Version header failed validation: Notion-Version header should be defined."); + } + if (!parsed.ok) return error(400, "invalid_json", "Error parsing JSON body."); + const body = (parsed.value ?? {}) as Record; + + const segments = path.split("/").filter(Boolean); + const [resource, rawId, extra] = segments; + if (extra !== undefined || !resource) return invalidUrl(method, path); + const id = rawId === undefined ? null : normalizeId(rawId); + if (rawId !== undefined && !id) return error(400, "validation_error", `path failed validation: path.id should be a valid uuid, instead was \`"${rawId}"\`.`); + + if (resource === "pages") { + if (method === "GET" && id) return getPage(id); + if (method === "POST" && !id) return createPage(body); + if (method === "PATCH" && id) return updatePage(id, body); + } + if (resource === "databases") { + if (method === "GET" && id) return getDatabase(id); + if (method === "POST" && !id) return createDatabase(body); + if (method === "PATCH" && id) return updateDatabase(id, body); + } + return invalidUrl(method, path); + } + + // ── Pages ────────────────────────────────────────────────────────── + + function getPage(id: string): NotionFakeResponse { + const page = pages.get(id); + if (!page) return notFound("page", id); + return ok(page); + } + + function createPage(body: Record): NotionFakeResponse { + const parent = (body.parent ?? {}) as { database_id?: unknown; page_id?: unknown }; + if (typeof parent.database_id !== "string") { + return error(400, "validation_error", "body failed validation: body.parent.database_id should be defined."); + } + const databaseId = normalizeId(parent.database_id); + const database = databaseId ? databases.get(databaseId) : undefined; + if (!database || database.archived || database.in_trash) return notFound("database", parent.database_id); + + const checked = checkValues(database, body.properties, true); + if ("error" in checked) return checked.error; + + const at = new Date().toISOString(); + const page: FakePage = { + object: "page", + id: randomId(), + parent: { type: "database_id", database_id: database.id }, + properties: checked.properties, + archived: false, + in_trash: false, + created_time: at, + last_edited_time: at, + }; + pages.set(page.id, page); + return ok(page); + } + + function updatePage(id: string, body: Record): NotionFakeResponse { + const page = pages.get(id); + if (!page) return notFound("page", id); + + const archiving = typeof body.archived === "boolean" ? body.archived : typeof body.in_trash === "boolean" ? body.in_trash : undefined; + const staysArchived = archiving === undefined ? page.archived : archiving; + if (body.properties !== undefined && staysArchived) { + return error(400, "validation_error", "Can't edit block that is archived. You must unarchive the block before editing."); + } + + if (body.properties !== undefined) { + if (page.parent.type !== "database_id") { + return error(400, "validation_error", "This fake only edits properties of pages in a database."); + } + const database = databases.get(page.parent.database_id); + if (!database) return notFound("database", page.parent.database_id); + const checked = checkValues(database, body.properties, false); + if ("error" in checked) return checked.error; + page.properties = { ...page.properties, ...checked.properties }; + } + if (archiving !== undefined) { + page.archived = archiving; + page.in_trash = archiving; + } + page.last_edited_time = new Date().toISOString(); + return ok(page); + } + + /** + * Values against the database's schema, keyed by property name. `create` + * also fills an empty value for every schema property the body omitted, as + * Notion's page object does. + */ + function checkValues( + database: FakeDatabase, + raw: unknown, + create: boolean + ): { properties: FakePage["properties"] } | { error: NotionFakeResponse } { + if (raw !== undefined && (raw === null || typeof raw !== "object" || Array.isArray(raw))) { + return { error: error(400, "validation_error", "body failed validation: body.properties should be an object.") }; + } + const values = (raw ?? {}) as Record; + const out: FakePage["properties"] = {}; + for (const [name, value] of Object.entries(values)) { + const property = database.properties[name]; + if (!property) { + return { error: error(400, "validation_error", `${name} is not a property that exists.`) }; + } + if (!value || typeof value !== "object" || Array.isArray(value)) { + return { error: error(400, "validation_error", `${name} is expected to be ${property.type}.`) }; + } + const record = value as Record; + const declared = typeof record.type === "string" ? record.type : Object.keys(record).find((key) => PROPERTY_TYPES.has(key)); + if (declared !== property.type || !(property.type in record)) { + return { error: error(400, "validation_error", `${name} is expected to be ${property.type}.`) }; + } + out[name] = { id: property.id, type: property.type, [property.type]: normalizeValue(property.type, record[property.type]) }; + } + if (create) { + for (const property of Object.values(database.properties)) { + if (!(property.name in out)) out[property.name] = { id: property.id, type: property.type, [property.type]: emptyValue(property.type) }; + } + } + return { properties: out }; + } + + // ── Databases ────────────────────────────────────────────────────── + + function getDatabase(id: string): NotionFakeResponse { + const database = databases.get(id); + if (!database) return notFound("database", id); + return ok(database); + } + + function createDatabase(body: Record): NotionFakeResponse { + const parent = (body.parent ?? {}) as { page_id?: unknown }; + if (typeof parent.page_id !== "string") { + return error(400, "validation_error", "body failed validation: body.parent.page_id should be defined."); + } + const pageId = normalizeId(parent.page_id); + const page = pageId ? pages.get(pageId) : undefined; + if (!page || page.archived || page.in_trash) return notFound("page", parent.page_id); + + const built = buildProperties(body.properties, {}); + if ("error" in built) return built.error; + const titles = Object.values(built.properties).filter((property) => property.type === "title"); + if (titles.length !== 1) { + return error(400, "validation_error", "Title is not provided or there are multiple title properties."); + } + + const at = new Date().toISOString(); + const database: FakeDatabase = { + object: "database", + id: randomId(), + parent: { type: "page_id", page_id: page.id }, + title: normalizeRichTextArray(body.title), + description: normalizeRichTextArray(body.description), + properties: built.properties, + archived: false, + in_trash: false, + created_time: at, + last_edited_time: at, + }; + databases.set(database.id, database); + return ok(database); + } + + function updateDatabase(id: string, body: Record): NotionFakeResponse { + const database = databases.get(id); + if (!database) return notFound("database", id); + if (body.properties !== undefined) { + const built = buildProperties(body.properties, database.properties); + if ("error" in built) return built.error; + database.properties = built.properties; + } + if (body.title !== undefined) database.title = normalizeRichTextArray(body.title); + if (body.description !== undefined) database.description = normalizeRichTextArray(body.description); + if (typeof body.archived === "boolean") { + database.archived = body.archived; + database.in_trash = body.archived; + } + database.last_edited_time = new Date().toISOString(); + return ok(database); + } + + /** + * A property schema from a request's `properties`, merged over `existing`: + * `{ Name: { title: {} } }` becomes `{ Name: { id, name: "Name", type: "title", title: {} } }`. + * `null` removes a property; `{ name }` renames one. + */ + function buildProperties( + raw: unknown, + existing: Record + ): { properties: Record } | { error: NotionFakeResponse } { + if (raw === null || typeof raw !== "object" || Array.isArray(raw)) { + return { error: error(400, "validation_error", "body failed validation: body.properties should be an object.") }; + } + const out: Record = { ...existing }; + for (const [name, value] of Object.entries(raw as Record)) { + if (value === null) { + delete out[name]; + continue; + } + if (typeof value !== "object" || Array.isArray(value)) { + return { error: error(400, "validation_error", `body.properties.${name} should be an object.`) }; + } + const spec = value as Record; + const current = out[name]; + const type = typeof spec.type === "string" ? spec.type : Object.keys(spec).find((key) => PROPERTY_TYPES.has(key)) ?? current?.type; + if (!type || !PROPERTY_TYPES.has(type)) { + return { error: error(400, "validation_error", `body.properties.${name} has no property type.`) }; + } + const config = (spec[type] ?? current?.[type] ?? {}) as Record; + if (type === "relation") { + const target = typeof config.database_id === "string" ? normalizeId(config.database_id) : null; + if (!target || !databases.has(target)) { + return { error: error(400, "validation_error", `body.properties.${name}.relation.database_id should be a database the integration can access.`) }; + } + config.database_id = target; + } + const renamed = typeof spec.name === "string" && spec.name ? spec.name : name; + if (renamed !== name) delete out[name]; + out[renamed] = { id: current?.id ?? propertyId(), name: renamed, type, [type]: config }; + } + return { properties: out }; + } + + seed(); + + return { handle, databases, pages, requests, failNext, addPage, pagesIn, reset }; +} + +// ── Helpers ────────────────────────────────────────────────────────── + +function ok(body: unknown): NotionFakeResponse { + return { status: 200, headers: { ...JSON_HEADERS }, body: clone(body) }; +} + +function error(status: number, code: string, message: string, extra: Record = {}): NotionFakeResponse { + return { status, headers: { ...JSON_HEADERS, ...extra }, body: { object: "error", status, code, message } }; +} + +function notFound(kind: "page" | "database", id: string): NotionFakeResponse { + return error( + 404, + "object_not_found", + `Could not find ${kind} with ID: ${id}. Make sure the relevant pages and databases are shared with your integration.` + ); +} + +function invalidUrl(method: string, path: string): NotionFakeResponse { + return error(400, "invalid_request_url", `Invalid request URL: ${method} ${path}.`); +} + +function codeForStatus(status: number): string { + if (status === 401) return "unauthorized"; + if (status === 403) return "restricted_resource"; + if (status === 404) return "object_not_found"; + if (status === 409) return "conflict_error"; + if (status === 429) return "rate_limited"; + if (status === 502) return "bad_gateway"; + if (status === 503) return "service_unavailable"; + if (status >= 500) return "internal_server_error"; + return "validation_error"; +} + +function header(headers: Record | Headers, name: string): string | null { + if (typeof (headers as Headers).get === "function") return (headers as Headers).get(name); + const record = headers as Record; + for (const [key, value] of Object.entries(record)) { + if (key.toLowerCase() === name) return value; + } + return null; +} + +function normalizePath(path: string): string { + const bare = path.split("?")[0].replace(/\/+$/, ""); + const withSlash = bare.startsWith("/") ? bare : `/${bare}`; + return withSlash.startsWith("/v1/") ? withSlash.slice(3) : withSlash; +} + +function parseBody(body: unknown): { ok: true; value: unknown } | { ok: false } { + if (body === undefined || body === null || body === "") return { ok: true, value: undefined }; + if (typeof body !== "string") return { ok: true, value: body }; + try { + return { ok: true, value: JSON.parse(body) }; + } catch { + return { ok: false }; + } +} + +/** A Notion id — 32 hex, dashed or not — as a dashed lower-case uuid; null otherwise. */ +function normalizeId(value: string): string | null { + const hex = value.replace(/-/g, "").toLowerCase(); + if (!/^[0-9a-f]{32}$/.test(hex)) return null; + return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20)}`; +} + +function randomId(): string { + return crypto.randomUUID(); +} + +function propertyId(): string { + return Math.random().toString(36).slice(2, 6); +} + +function richText(content: string): RichText { + return { type: "text", text: { content }, plain_text: content }; +} + +function normalizeRichTextArray(value: unknown): RichText[] { + if (!Array.isArray(value)) return []; + return value.map((item) => { + const part = (item ?? {}) as RichText; + return { type: "text", ...part, plain_text: part.plain_text ?? part.text?.content ?? "" }; + }); +} + +function normalizeValue(type: string, value: unknown): unknown { + if (type === "title" || type === "rich_text") return normalizeRichTextArray(value); + return clone(value); +} + +function emptyValue(type: string): unknown { + if (type === "title" || type === "rich_text" || type === "multi_select" || type === "people" || type === "files" || type === "relation") return []; + if (type === "checkbox") return false; + return null; +} + +function clone(value: T): T { + return value === undefined ? value : (JSON.parse(JSON.stringify(value)) as T); +} diff --git a/v5/test/msw/notion-mirror.ts b/v5/test/msw/notion-mirror.ts new file mode 100644 index 0000000..464e016 --- /dev/null +++ b/v5/test/msw/notion-mirror.ts @@ -0,0 +1,29 @@ +import { http, HttpResponse } from "msw"; +import type { SetupServer } from "msw/node"; +import type { NotionFake } from "../fakes/notion-fake.ts"; + +/** + * Install a {@link NotionFake} as MSW handlers (spec §10: "MSW for Notion"). + * + * Every method on `${base}/*` goes to `fake.handle`, which answers exactly as + * the E2E stub server does, so a push tested here and a push clicked through + * in Playwright meet the same Notion. Registered with `server.use`, so the + * shared `afterEach(resetHandlers)` in `vitest.setup.ts` removes it; call this + * in a `beforeEach` (or in the test) rather than once per file. + * + * The request body is passed as text and parsed by the fake; the headers are + * handed over for the token check and never logged. + */ +export function useNotionFake(server: Pick, fake: NotionFake, base = "https://api.notion.com/v1"): void { + const prefix = new URL(base).pathname.replace(/\/+$/, ""); + const trimmedBase = base.replace(/\/+$/, ""); + server.use( + http.all(`${trimmedBase}/*`, async ({ request }) => { + const url = new URL(request.url); + const path = url.pathname.startsWith(prefix) ? url.pathname.slice(prefix.length) : url.pathname; + const text = request.method === "GET" || request.method === "HEAD" ? "" : await request.text(); + const response = fake.handle(request.method, path, request.headers, text); + return HttpResponse.json(response.body as never, { status: response.status, headers: response.headers }); + }) + ); +} diff --git a/v5/vitest.setup.ts b/v5/vitest.setup.ts index 9b55d82..a96839d 100644 --- a/v5/vitest.setup.ts +++ b/v5/vitest.setup.ts @@ -2,6 +2,16 @@ import "@testing-library/jest-dom"; import { afterAll, afterEach, beforeAll, beforeEach, vi } from "vitest"; import { server } from "./test/msw/server"; +// ── Blob: "none" unless a test opts in ───────────────────────────── +// +// Outside Vercel, no `BLOB_READ_WRITE_TOKEN` means the `.blob-data/` folder +// (src/lib/blob-mode.ts). Tests keep the old meaning — no token, no store — so +// the `blob_not_configured` branches stay covered and no test writes to the +// working tree. A local-mode test stubs this to "" and points the store at a +// temporary folder. Set directly (not `vi.stubEnv`) so `unstubAllEnvs` after +// each test restores it rather than removing it. +process.env.BLOB_LOCAL_DISABLE ??= "1"; + // ── Web Storage shim ─────────────────────────────────────────────── // // Under this Node/jsdom combo `window.localStorage` is a bare object missing