Correct documentation claims the first pass did not audit

README.md

@@ -98,6 +98,13 @@ erDiagram
     Webhooks ||--o{ WebhookDeliveries : "tracks"
 ```
 
+Two further tables carry no foreign keys and so sit outside the diagram:
+
+- **`AnalyticsEvents`** — the durable journey event trail. Rows reference a session by its id as a
+  string rather than by FK, because events outlive nothing and are pruned on their own schedule.
+- **`SessionReadModels`** — a denormalised projection of a session used by the operator session
+  list, so that view does not join five tables per row.
+
 ### Table reference
 
 #### `Flows`
@@ -122,7 +129,7 @@ A single step in a flow. `JsonContent` carries type-specific configuration; `Com
 | `Id` | `uuid` | PK | Node identifier |
 | `FlowId` | `uuid` | FK → `Flows.Id` CASCADE | Owning flow |
 | `Key` | `varchar(100)` | NOT NULL, UNIQUE per flow | Stable string identifier (used in routing logic and URL interpolation) |
-| `Type` | `integer` | NOT NULL | Enum: `Form=0`, `Information=1`, `Logic=2`, `Redirect=3`, `DocumentUpload=4` |
+| `Type` | `integer` | NOT NULL | Enum: `Form=0`, `DocumentUpload=1`, `Redirect=2`, `Information=3`, `Logic=4`. Stored as the ordinal with no value conversion, so these integers are what is in the column. |
 | `Title` | `varchar(200)` | NOT NULL | Display title shown to the end user |
 | `JsonContent` | `text` | NOT NULL | Type-specific configuration JSON (field definitions, upload constraints, redirect URL, etc.) |
 | `ComplianceRuleJson` | `text` | nullable | Server-side validation rules (see [ComplianceRuleJson reference](#compliancerulejson-reference)) |
@@ -246,6 +253,58 @@ Immutable snapshots of a flow graph captured on each publish. Enables rollback a
 
 **Indexes:** `(FlowId, VersionNumber)` UNIQUE
 
+#### `AnalyticsEvents`
+
+The durable journey event trail. Added so the per-step record of where applicants stall could be
+queried and exported rather than only written to the application log. Read per session by
+`GET /api/analytics/sessions/{sessionId}/events`, and surfaced in the operator console under a
+session's **Event trail**.
+
+| Column | Type | Constraints | Description |
+|--------|------|-------------|-------------|
+| `Id` | `uuid` | PK | The originating event's id; doubles as the idempotency key for client retries |
+| `EventType` | `varchar(100)` | NOT NULL | For example `session_started`, `step_view`, `journey_complete` |
+| `JourneyId` | `varchar(100)` | NOT NULL | The flow the event belongs to |
+| `SessionId` | `varchar(100)` | NOT NULL | The session the event belongs to. A string, not an FK — events are pruned on their own schedule and must not depend on the session row surviving |
+| `StepId` | `varchar(100)` | nullable | Node the event relates to, when there is one |
+| `StepIndex` | `integer` | nullable | Zero-based position within the session's submission history |
+| `PayloadJson` | `text` | NOT NULL | Event-specific payload, so a new event type needs no schema change |
+| `Source` | `varchar(20)` | NOT NULL | `server` or `client`. Stamped by the API, never taken from the caller, so a client cannot pass its events off as server-raised |
+| `OccurredAt` | `timestamptz` | NOT NULL | When the event happened, as reported by its origin |
+| `RecordedAt` | `timestamptz` | NOT NULL | When the row was written. Retention is measured from this, not `OccurredAt`, which a client controls and could backdate |
+
+**Indexes:** `(SessionId, OccurredAt)` — reading one session's trail in order is the common query
+
+Pruned by a background sweep after `Analytics__RetentionDays` (default 365). Set
+`Analytics__DatabaseProvider__Enabled=false` to disable storage entirely; the application keeps
+working and events go only to the log.
+
+---
+
+#### `SessionReadModels`
+
+A denormalised projection of a session, maintained so the operator session list does not join five
+tables per row. Not a source of truth: every column is derived from `Sessions` and its relations.
+
+| Column | Type | Constraints | Description |
+|--------|------|-------------|-------------|
+| `Id` | `uuid` | PK | Mirrors `Sessions.Id` |
+| `FlowId` | `uuid` | NOT NULL | The flow being executed |
+| `FlowName` | `text` | NOT NULL | Denormalised from `Flows.Name` |
+| `CustomerEmail` | `text` | nullable | Denormalised from the linked `CustomerProfiles` row |
+| `CustomerCountry` | `text` | nullable | Denormalised from the linked `CustomerProfiles` row |
+| `ExternalCustomerId` | `text` | nullable | Denormalised from the linked `CustomerProfiles` row |
+| `CurrentNodeId` | `uuid` | nullable | Node awaiting submission; `null` once terminal |
+| `CurrentNodeKey` | `text` | nullable | Denormalised from `Nodes.Key` |
+| `CurrentNodeTitle` | `text` | nullable | Denormalised from `Nodes.Title` |
+| `StatusName` | `text` | NOT NULL | Status as a string, so the list needs no enum mapping |
+| `StepCount` | `integer` | NOT NULL | Number of submissions recorded |
+| `CreatedAt` | `timestamptz` | NOT NULL | Session creation timestamp |
+| `UpdatedAt` | `timestamptz` | NOT NULL | Last modification timestamp |
+| `CompletedAt` | `timestamptz` | nullable | Set when the session completes |
+
+---
+
 ### Design notes
 
 **All primary keys are UUIDs** — generated by the caller for flows and nodes (to support offline graph construction in the builder), generated server-side for runtime records (sessions, submissions, deliveries).

docs/journey-json.md

@@ -124,7 +124,7 @@ The journey JSON links directly to frontend behavior:
 | `connections[]` | Runtime engine + `JourneyBuilder.tsx` | Drives routing and shows labeled graph edges in the read-only session monitor. |
 | `nodes[]`/`connections[]` | `VisualJourneyBuilder.tsx` + `VisualJourneyCanvas.tsx` | Edited visually via drag-and-drop in the admin builder at `/admin/journey-builder`. |
 | `nodes[]`/`connections[]` | `FlowAuthoringPanel.tsx` + `flowAuthoring.ts` | Alternative JSON authoring panel for power users; runs alongside the visual builder. |
-| `nodes[].type` | `VisualJourneyCanvas.tsx` (`NODE_TYPE_STYLES`) | Determines the color scheme of each node on the canvas (Form=blue, DocumentUpload=purple, Redirect=amber, Information=green, Logic=orange). |
+| `nodes[].type` | `visualJourneyCanvasUtils.ts` (`NODE_TYPE_STYLES`) | Determines the color scheme of each node on the canvas (Form=blue, DocumentUpload=purple, Redirect=amber, Information=green, Logic=orange). |
 | `nodes[].isStartNode` | `VisualJourneyCanvas.tsx` + `NodePropertiesPanel.tsx` | Start node receives a bold outline; Properties Panel enforces at most one start node. |
 
 ## Journey visualization

docs/runbook.md

@@ -295,21 +295,41 @@ Docker Compose configures automatic restart when the liveness probe fails.
 
 The API exposes Prometheus metrics at `GET /metrics`.
 
-Key metrics to monitor:
-
-| Metric | Description | Alert threshold |
-|--------|-------------|-----------------|
-| `dotnet_duration_seconds` | HTTP request duration | p95 > 2s |
-| `waymark_session_started_total` | New sessions started | Sudden drop |
-| `waymark_session_completed_total` | Sessions completed | Sudden drop |
-| `waymark_session_abandoned_total` | Sessions abandoned by timeout | Spike > normal |
-| `waymark_webhook_delivery_failed_total` | Failed webhook deliveries | > 5 in 5 min |
-| `waymark_webhook_delivery_retried_total` | Webhook retries | Sustained high rate |
-| `process_cpu_seconds_total` | Process CPU usage | > 80% for 5 min |
-
-**Grafana dashboard** (if using Grafana Cloud / self-hosted):
-- Import the provided dashboard JSON from `docs/grafana-dashboard.json` (if available)
-- Data source: Prometheus scraping `http://api:8080/metrics` every 15s
+The endpoint requires authentication and is exempt from rate limiting.
+
+### Application metrics
+
+These are the metrics the application emits. Names are as registered in
+`PrometheusMetricsService`; build alerts against these exact strings.
+
+| Metric | Type | Labels | Emitted when | Alert on |
+|--------|------|--------|--------------|----------|
+| `onboarding_sessions_started_total` | counter | `flowId` | A session starts | Sudden drop |
+| `onboarding_sessions_completed_total` | counter | `flowId` | A session completes | Sudden drop, or a widening gap against started |
+| `onboarding_webhook_deliveries_total` | counter | `status` (`delivered`, `failed`) | Each delivery attempt resolves | `failed` rate > 5 in 5 min |
+| `waymark_virus_scan_bypassed_total` | counter | — | A document upload is accepted without a real scan | Any increase outside a deliberately unscanned environment |
+
+`prometheus-net` also exports standard .NET runtime and process metrics on the same endpoint, among
+them `process_cpu_seconds_total` (alert above 80% for 5 min) and the GC and thread-pool series.
+
+### What is not exposed
+
+Worth knowing before you design a dashboard around metrics that will never arrive:
+
+- **HTTP request duration and status code counts.** `UseHttpMetrics()` is not wired up, so there is
+  no `http_request_duration_seconds` series and no per-endpoint latency or error rate. Request-level
+  monitoring has to come from your ingress or APM until that middleware is added.
+- **Abandoned sessions.** There is no counter for sessions the timeout sweep abandons. Track
+  abandonment from the `Sessions` table or the analytics event trail instead.
+- **`onboarding_active_sessions`.** The gauge is registered but nothing ever sets it, so it reads
+  zero permanently. Do not alert on it.
+
+### Grafana
+
+There is no dashboard JSON in this repository; build one against the metric names above.
+
+- Data source: Prometheus scraping `http://api:8080/metrics` every 15s.
+- The scraper needs a credential, since `/metrics` requires authentication.
 
 ---
 

website/docs/test-journeys.md

@@ -28,21 +28,21 @@ Each journey isolates a distinct capability of the platform.
 ```bash
 # 1. Start session
 curl -X POST http://localhost:5072/api/workflow/sessions/start \
-  -H 'X-Api-Key: dev-key' \
+  -H 'X-Api-Key: dev-api-key-change-in-production' \
   -H 'Content-Type: application/json' \
   -d '{"flowId":"a1a1a1a1-a1a1-a1a1-a1a1-a1a1a1a1a1a1"}'
 
 # 2. Submit form (valid)
 curl -X POST http://localhost:5072/api/workflow/sessions/{sessionId}/steps/{nodeId}/submit \
-  -H 'X-Api-Key: dev-key' \
+  -H 'X-Api-Key: dev-api-key-change-in-production' \
   -H 'Content-Type: application/json' \
-  -d '{"FullName":"Alice Example","Email":"alice@example.com"}'
+  -d '{"payload":{"FullName":"Alice Example","Email":"alice@example.com"}}'
 
 # 3. Submit Information node (empty payload advances session)
 curl -X POST http://localhost:5072/api/workflow/sessions/{sessionId}/steps/{nodeId}/submit \
-  -H 'X-Api-Key: dev-key' \
+  -H 'X-Api-Key: dev-api-key-change-in-production' \
   -H 'Content-Type: application/json' \
-  -d '{}'
+  -d '{"payload":{}}'
 # → {"isCompleted":true,"currentNode":null}
 ```
 
@@ -77,11 +77,11 @@ See [`src/frontend/src/schemas/journey-a-linear-basic.json`](../../src/frontend/
 
 ```bash
 # France → EU disclosure
-curl -X POST .../steps/{nodeId}/submit -d '{"Country":"France"}'
+curl -X POST .../steps/{nodeId}/submit -d '{"payload":{"Country":"France"}}'
 # → currentNode.key = "eu-gdpr-disclosure"
 
 # USA → global terms
-curl -X POST .../steps/{nodeId}/submit -d '{"Country":"USA"}'
+curl -X POST .../steps/{nodeId}/submit -d '{"payload":{"Country":"USA"}}'
 # → currentNode.key = "global-terms-and-conditions"
 ```
 
@@ -126,15 +126,15 @@ The raw `{{sessionId}}` token is **never** sent to the frontend.
 curl -X POST .../sessions/start -d '{"flowId":"c3c3c3c3-..."}'
 
 # 2a. Invalid NationalId → 422
-curl -X POST .../steps/{nodeId}/submit -d '{"NationalId":"INVALID"}'
+curl -X POST .../steps/{nodeId}/submit -d '{"payload":{"NationalId":"INVALID"}}'
 # → 422 {"violations":[{"field":"NationalId","message":"..."}]}
 
 # 2b. Valid NationalId → advances to DocumentUpload
-curl -X POST .../steps/{nodeId}/submit -d '{"NationalId":"AB123456"}'
+curl -X POST .../steps/{nodeId}/submit -d '{"payload":{"NationalId":"AB123456"}}'
 # → currentNode.type = "DocumentUpload"
 
 # 3. DocumentUpload submission (empty payload in test mode skips virus scan)
-curl -X POST .../steps/{nodeId}/submit -d '{}'
+curl -X POST .../steps/{nodeId}/submit -d '{"payload":{}}'
 # → currentNode.type = "Redirect", jsonContent.url contains real sessionId
 ```
 

website/index.html

@@ -57,7 +57,7 @@ <h1 class="text-5xl md:text-7xl font-extrabold tracking-tight text-white leading
       </h1>
       <p class="text-xl text-slate-400 max-w-2xl mx-auto mb-10 leading-relaxed">
         Waymark decouples journey logic from your frontend and backend code.
-        Define states, transitions, and conditions in structured JSON or YAML —
+        Define states, transitions, and conditions as structured JSON —
         and let the engine handle the rest.
       </p>
       <div class="flex flex-col sm:flex-row items-center justify-center gap-4">
@@ -319,7 +319,7 @@ <h2 class="text-3xl md:text-4xl font-bold text-white mb-4">Everything you need t
             <svg class="w-6 h-6" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"><path d="M9 3H5a2 2 0 00-2 2v4m6-6h10a2 2 0 012 2v4M9 3v18m0 0h10a2 2 0 002-2V9M9 21H5a2 2 0 01-2-2V9m0 0h18"/></svg>
           </div>
           <h3 class="font-semibold text-white mb-2 group-hover:text-brand-400 transition-colors">Schema-First Design</h3>
-          <p class="text-sm text-slate-400 leading-relaxed">Define states, transitions, and conditions in JSON or YAML. The engine interprets your schema at runtime — no compilation required.</p>
+          <p class="text-sm text-slate-400 leading-relaxed">Define states, transitions, and conditions as JSON. The engine interprets your schema at runtime — no compilation required.</p>
         </div>
 
         <div class="feature-card rounded-xl border border-slate-800 bg-slate-900 p-6 hover:border-violet-500/50 hover:shadow-xl hover:shadow-violet-500/10 transition-all duration-300 group">
@@ -351,7 +351,7 @@ <h3 class="font-semibold text-white mb-2 group-hover:text-amber-400 transition-c
             <svg class="w-6 h-6" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"><path d="M7 16a4 4 0 01-.88-7.903A5 5 0 1115.9 6L16 6a5 5 0 011 9.9M15 13l-3-3m0 0l-3 3m3-3v12"/></svg>
           </div>
           <h3 class="font-semibold text-white mb-2 group-hover:text-sky-400 transition-colors">Document Upload</h3>
-          <p class="text-sm text-slate-400 leading-relaxed">Built-in <code class="text-xs bg-slate-800 px-1 rounded">DocumentUpload</code> node type with file type filtering, size limits, and progress streaming via SSE.</p>
+          <p class="text-sm text-slate-400 leading-relaxed">Built-in <code class="text-xs bg-slate-800 px-1 rounded">DocumentUpload</code> node type with file type filtering, size limits, per-file upload progress, and optional virus scanning.</p>
         </div>
 
         <div class="feature-card rounded-xl border border-slate-800 bg-slate-900 p-6 hover:border-rose-500/50 hover:shadow-xl hover:shadow-rose-500/10 transition-all duration-300 group">