v5 data platform, Phase 8: the Notion mirror - #39
Merged
Merged
Conversation
…wn Notion /admin/mirror connects a Notion integration token (validated by a read, stored AES-256-GCM under a key derived from AUTH_SECRET), creates or validates seven databases, and shows the last sync, status and error. The mirrorPush workflow pushes changed rows in dependency order, in 45 s rounds, throttled to 3 req/s with Retry-After; partial failure never advances last_synced_at, a 401 pauses the mirror. Approvals, editor writes, project publishing and ticket changes request a coalesced push; the daily cron is the backstop. Per the 2026-09-23 decision, reporter and author names and emails are mirrored. A mapping change fences off a running push, a reset re-pushes the pages that link to it, and a demoted or banned owner's mirror stops pushing. Migration 0007. E2E 8 runs against a stateful Notion fake behind a test-only NOTION_API_BASE_URL. Gate: lint 0 errors, 194 files / 2486 tests, 69 Playwright, spec:coverage 0 undocumented, build green with /admin/mirror Partial Prerender. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DNV63U6ERy45Gj2TThBfyC
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Note
Copilot was unable to run its full agentic suite in this review.
Copilot review overview
Review effort: Lite
Findings: 4
Open (6)
Date.parse()can returnNaNfor an unexpected/invalid ISO string; sinceNaN !== null, this… · Newconsole.*but doesn’t restore them afterward. That can leak… · Newconsole.*but doesn’t restore them afterward. That can leak… · NewHttpResponse.json(...)forces a JSON response even when the fake intends an empty body (e.g.… · Newpollingis… · Newpollingis… · New
What changed in this PR
Implements Phase 8 of the v5 data platform spec: an owner-only Notion “mirror” with secure token storage, admin UI to connect/map/control it, workflow-driven pushes, and cron + app triggers to keep it up to date.
Changes:
- Adds mirror domain model/migrations plus data-layer helpers for mapping and page tracking.
- Implements workflows/steps/triggers/backstop to push to Notion with coalescing, retries, and safety fences.
- Adds
/admin/mirrorUI (connect, mapping, status, controls), test infrastructure (MSW + stub server), and extensive unit/integration/E2E coverage.
| File | Description |
|---|---|
| v5/test/msw/notion-mirror.ts | MSW adapter to route Notion API calls into the in-memory Notion fake |
| v5/src/workflows/mirror-push.workflow.test.ts | In-process workflow-tier test for mirrorPush using real runtime + seeded DB + MSW fake |
| v5/src/workflows/mirror-push.ts | Workflow orchestration for push rounds and coalesced pushes |
| v5/src/workflows/mirror-push.test.ts | Unit tests for workflow orchestration logic with mocked steps/sleep |
| v5/src/styles/globals.css | Removes .admin-index-note styling (admin index note removed) |
| v5/src/lib/rate-limit.ts | Adds MIRROR_SETUP_TIER for mirror setup rate limiting |
| v5/src/lib/mirror/types.ts | Client-safe mirror types and error/setup vocabularies |
| v5/src/lib/mirror/trigger.ts | requestMirrorPush() coalescing trigger that never throws |
| v5/src/lib/mirror/trigger.test.ts | PGlite tests for trigger coalescing and failure swallowing |
| v5/src/lib/mirror/token-crypto.ts | AES-256-GCM + HKDF token encryption/decryption |
| v5/src/lib/mirror/token-crypto.test.ts | Tests for token crypto round-trips, rotation failures, scrubbing guarantees |
| v5/src/lib/mirror/steps.ts | Workflow steps + database-error classification into retryable/fatal |
| v5/src/lib/mirror/steps.test.ts | Direct tests for step behavior and error classification/scrubbing |
| v5/src/lib/mirror/start.ts | Starts workflows and implements syncMirrorNow claim/release logic |
| v5/src/lib/mirror/start.test.ts | PGlite tests for Sync Now window/claims and workflow start contracts |
| v5/src/lib/mirror/owner-roles.ts | Hardcoded stored roles allowed to own/push mirrors |
| v5/src/lib/mirror/owner-roles.test.ts | Pins MIRROR_OWNER_ROLES to can(..., "mirror.manage") |
| v5/src/lib/mirror/notion-id.ts | Parses Notion page/db IDs from pasted IDs/URLs |
| v5/src/lib/mirror/notion-id.test.ts | Tests for Notion ID parsing cases |
| v5/src/lib/mirror/limits.ts | Central constants for mirror budgets, windows, coalescing, polling |
| v5/src/lib/mirror/database-schemas.test.ts | Tests fixed database schemas and schema validation |
| v5/src/lib/mirror/credentials.ts | Decrypts stored token exactly once to build Notion client |
| v5/src/lib/mirror/credentials.test.ts | Tests client creation uses decrypted token and honors options |
| v5/src/lib/mirror/connect.ts | Test/connect logic that validates token+page before storing |
| v5/src/lib/mirror/connect.test.ts | Fake-Notion-backed tests for connect/test behaviors and non-leakage |
| v5/src/lib/intake/approve.ts | Adds mirror trigger after approve/add-unit commit |
| v5/src/lib/intake/approve.test.ts | Verifies mirror trigger call behavior for approvals/units |
| v5/src/lib/db/schema/vocabulary.ts | Adds MIRROR_ENTITY and MIRROR_STATUS vocabularies |
| v5/src/lib/db/schema/mirror.ts | Adds notion_mirrors / mirror_pages schema + bytea helpers |
| v5/src/lib/db/schema/mirror.test.ts | Migration-level constraints/cascades/trigger/bytea tests |
| v5/src/lib/db/schema/index.ts | Exports mirror schema, updates schema module doc |
| v5/src/lib/db/migrations/meta/_journal.json | Registers migration 0007_notion_mirror |
| v5/src/lib/db/migrations/0007_notion_mirror.sql | Creates mirror tables and updated_at trigger |
| v5/src/lib/data/users.ts | Updates email-handling comment to reflect mirror amendment |
| v5/src/lib/data/mirror-pages.ts | Data access for mirror_pages (lookup/upsert/orphans) |
| v5/src/lib/data/mirror-pages.test.ts | PGlite tests for mirror_pages helpers and anti-join logic |
| v5/src/lib/data/maintenance.ts | Clarifies email selection boundaries vs mirror source selection |
| v5/src/lib/cron/mirror-backstop.ts | Daily cron stage to start pushes for mirrors behind their data |
| v5/src/lib/cron/mirror-backstop.test.ts | PGlite tests for backstop start/count behavior |
| v5/src/lib/cron/backup-policy.ts | Redacts notion_mirrors.tokenCiphertext in backups |
| v5/src/lib/cron/backup-policy.test.ts | Tests backup redaction for mirror token ciphertext |
| v5/src/components/admin/use-refresh-nudge.ts | Workaround hook to force commit of refreshed render post-action |
| v5/src/components/admin/use-refresh-nudge.test.tsx | Tests nudge timers/render behavior + cleanup on unmount |
| v5/src/components/admin/mirror-messages.test.ts | Ensures message keys exist for all mirror codes/entities/statuses |
| v5/src/components/admin/MirrorStatus.tsx | Mirror status UI panel with conditional polling |
| v5/src/components/admin/MirrorStatus.test.tsx | Tests status rendering, error display, and polling behavior |
| v5/src/components/admin/MirrorMapping.test.tsx | Tests mapping UI behavior and validation messaging |
| v5/src/components/admin/MirrorControls.tsx | Sync/Pause/Disconnect UI with user feedback + refresh nudge |
| v5/src/components/admin/MirrorControls.test.tsx | Tests control enablement/refusals/confirm flow and messaging |
| v5/src/components/admin/MirrorConnect.tsx | Connect/Test form UI with token handling and refresh nudge |
| v5/src/components/admin/MirrorConnect.test.tsx | Tests connect form UX and refusal/success handling |
| v5/src/app/api/cron/daily/route.ts | Adds mirror backstop stage to daily cron route |
| v5/src/app/api/cron/daily/route.test.ts | Tests mirror cron stage reporting and failure semantics |
| v5/src/app/admin/projects/actions.ts | Triggers mirror push on publish/unpublish |
| v5/src/app/admin/projects/actions.mirror.test.ts | Tests mirror trigger called only on successful publish/unpublish |
| v5/src/app/admin/page.tsx | Adds /admin/mirror surface and removes “more coming” note |
| v5/src/app/admin/mirror/page.tsx | New admin mirror page with owner-only enforcement and state machine |
| v5/src/app/admin/mirror/action-result.ts | Defines mirror action result shapes + message-key mapping |
| v5/src/app/admin/maintenance/actions.ts | Triggers mirror push after ticket updates commit |
| v5/src/app/admin/maintenance/actions.mirror.test.ts | Tests mirror trigger call discipline for ticket actions |
| v5/src/app/admin/inventory/tool-write-context.ts | Triggers mirror push after successful tool-editor writes |
| v5/src/app/admin/inventory/tool-write-context.mirror.test.ts | Tests tool-editor trigger wiring + “only on commit” behavior |
| v5/scripts/check-spec-coverage.ts | Accepts NOTION_API_BASE_URL as test-only env var |
| v5/playwright.config.ts | Adds Notion stub webServer + mirror Playwright project sequencing |
| v5/messages/en.json | Adds mirror UI/messages keys; removes “mirror later phase” note |
| v5/e2e/stubs/notion-stub.ts | HTTP stub server using the in-memory Notion fake for E2E |
| v5/e2e/stubs/notion-fixture.ts | Shared E2E Notion stub token/page fixtures |
| v5/e2e/mirror.spec.ts | End-to-end scenario for connect → create DBs → sync → disconnect |
| v5/TESTING.md | Documents mirror testing strategy (MSW fake + E2E stub) |
| docs/specs/2026-09-14-v5-data-platform-design.md | Adds 2026-09-23 as-built amendment and decisions/gaps |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+80
to
+86
| const mapped = Object.keys(view.mapping).length > 0; | ||
| const availableAt = view.syncAvailableAt ? Date.parse(view.syncAvailableAt) : null; | ||
| // Before hydration there is no clock, so the server's own answer stands: a | ||
| // `syncAvailableAt` at all means "not yet". | ||
| const windowClosed = availableAt !== null && (now === null || availableAt > now); | ||
| const minutesLeft = | ||
| availableAt !== null && now !== null ? Math.max(1, Math.ceil((availableAt - now) / 60_000)) : null; |
| vi.stubEnv("DATABASE_URL", ""); | ||
| await getDb(); | ||
| }); | ||
|
|
Comment on lines
+44
to
+49
| const printed: unknown[][] = []; | ||
| for (const method of ["log", "info", "warn", "error", "debug"] as const) { | ||
| vi.spyOn(console, method).mockImplementation((...args: unknown[]) => { | ||
| printed.push(args); | ||
| }); | ||
| } |
Comment on lines
+21
to
+27
| http.all(`${trimmedBase}/*`, async ({ request }) => { | ||
| const url = new URL(request.url); | ||
| const path = url.pathname.startsWith(prefix) ? url.pathname.slice(prefix.length) : url.pathname; | ||
| const text = request.method === "GET" || request.method === "HEAD" ? "" : await request.text(); | ||
| const response = fake.handle(request.method, path, request.headers, text); | ||
| return HttpResponse.json(response.body as never, { status: response.status, headers: response.headers }); | ||
| }) |
Comment on lines
+23
to
+26
| * **It polls while a push is running or requested, and only then** — as | ||
| * `IntakeList` does. The push happens in a workflow the page cannot subscribe | ||
| * to, so it asks for a fresh render every `MIRROR_POLL_INTERVAL_MS`; the | ||
| * interval is cleared the moment neither is true, and on unmount. |
Comment on lines
+45
to
+50
| const polling = view.running || view.syncPending; | ||
| useEffect(() => { | ||
| if (!polling) return; | ||
| const timer = setInterval(() => router.refresh(), MIRROR_POLL_INTERVAL_MS); | ||
| return () => clearInterval(timer); | ||
| }, [polling, router]); |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 90f2f78a45
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| body: body === undefined ? undefined : JSON.stringify(body), | ||
| // Deliberately not the remaining budget (see the header): a started | ||
| // request runs to Notion's answer or this ceiling. | ||
| signal: AbortSignal.timeout(NOTION_REQUEST_TIMEOUT_MS), |
There was a problem hiding this comment.
When a POST /pages reaches Notion but its response takes longer than 30 seconds, this signal aborts the local fetch even though Notion may already have created the page. The push consequently never records its ID in mirror_pages, so the next trigger creates another page for the same row; repeated timeouts can corrupt the mirror with duplicates. Creation requests need reconciliation or another mechanism that avoids retrying an ambiguously completed create.
AGENTS.md reference: v5/AGENTS.md:L388-L395
Useful? React with 👍 / 👎.
| } | ||
|
|
||
| /** `tool_id`, or null when that tool is archived. */ | ||
| const LIVE_TOOL_ID = sql`(case when t.archived_at is null then s.tool_id::text else null end)`; |
There was a problem hiding this comment.
When a previously mirrored tool is archived, this expression would correctly clear the Tool relation, but pageOf only selects a unit when the unit's own updated_at advanced. Archiving updates the tool row, not its units, resources, or maintenance logs, so those pages are never rebuilt and retain relations to the archived Notion page indefinitely; published projects have the same issue with their tool list. Mark relation dependents stale or include the joined tool revision in change detection when archive state changes.
AGENTS.md reference: v5/AGENTS.md:L411-L416
Useful? React with 👍 / 👎.
Comment on lines
+149
to
+152
| let fresh: NotionDatabaseObject; | ||
| try { | ||
| fresh = await client.createDatabase(databaseCreateBody(entity, parentPageId, resolved)); | ||
| } catch (error) { |
There was a problem hiding this comment.
If two Create databases actions run concurrently, both read the same missing mapping in openMirror, both execute this external create for every entity, and only persist their separate mappings after all creates finish. The last save wins while the other set of seven databases remains orphaned in the user's workspace. This can happen from two tabs or rapid duplicate submissions, so database setup needs a per-mirror claim or conditional reservation before issuing the Notion creates.
AGENTS.md reference: v5/AGENTS.md:L384-L387
Useful? React with 👍 / 👎.
| const format = useFormatter(); | ||
| const router = useRouter(); | ||
|
|
||
| const polling = view.running || view.syncPending; |
There was a problem hiding this comment.
When the page renders with pushScheduled true but the two-minute coalesced workflow has not claimed the mirror yet, neither condition here starts polling. The workflow can later run and finish successfully, but the open page continues to display “scheduled” and the old status indefinitely until an unrelated refresh. Include view.pushScheduled in the polling condition so this advertised pending state can transition on screen.
AGENTS.md reference: v5/AGENTS.md:L417-L424
Useful? React with 👍 / 👎.
…h edges Manual PDFs are archived into Blob (manuals/<tool>/<resource>.pdf) on approval, on editor link changes and from MCP create_tool, through a small workflow; the daily cron archives ten un-archived manuals a night, which also backfills the imported inventory. HTML pages, oversize files and repeats are refused. The tool page and chat prefer the archived copy. With no BLOB_READ_WRITE_TOKEN, next dev now stores files in the untracked .blob-data/ folder and serves public ones from /api/dev-blob/…; on Vercel or a production build nothing changes. BLOB_LOCAL_DISABLE keeps tests on "none". Tool descriptions render as Markdown, so researched specs show as a list. The intake card's checkboxes say what they do once the table stacks in the narrow chat panel. Gate: typecheck clean, lint 0 errors, 200 files / 2557 unit tests, workflow tier green, spec:coverage 0 undocumented. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DNV63U6ERy45Gj2TThBfyC
philosophercode
force-pushed
the
v5/data-platform-phase-8
branch
from
September 23, 2026 15:52
7aef424 to
0491174
Compare
philosophercode
changed the base branch from
v5/data-platform-phase-6
to
main
September 24, 2026 01:08
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DNV63U6ERy45Gj2TThBfyC
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #38 (Phase 6). Implements §3.8 of the data-platform spec; as-built details are in the second 2026-09-23 amendment.
What changes
/admin/mirror(owner-only,mirror.manage):mirrorPushworkflow oversrc/lib/mirror/*:Retry-Afterhonoured on 429.last_synced_atdoes not advance.requestMirrorPush(), which coalesces bursts and never fails the write it follows. The daily cron is the backstop.AUTH_SECRET, and never appears in logs, errors or the backup.0007_notion_mirror.sql.Tests
NOTION_API_BASE_URL.Still to do
useRefreshNudgeworks around a stall: a refresh after a server action was rendered but not shown in the production build. The root cause (Next 16.1 / React 19.2) is not fixed.🤖 Generated with Claude Code
https://claude.ai/code/session_01DNV63U6ERy45Gj2TThBfyC