diff --git a/README.md b/README.md index df2ed77..db27045 100644 --- a/README.md +++ b/README.md @@ -98,6 +98,13 @@ erDiagram Webhooks ||--o{ WebhookDeliveries : "tracks" ``` +Two further tables carry no foreign keys and so sit outside the diagram: + +- **`AnalyticsEvents`** — the durable journey event trail. Rows reference a session by its id as a + string rather than by FK, because events outlive nothing and are pruned on their own schedule. +- **`SessionReadModels`** — a denormalised projection of a session used by the operator session + list, so that view does not join five tables per row. + ### Table reference #### `Flows` @@ -122,7 +129,7 @@ A single step in a flow. `JsonContent` carries type-specific configuration; `Com | `Id` | `uuid` | PK | Node identifier | | `FlowId` | `uuid` | FK → `Flows.Id` CASCADE | Owning flow | | `Key` | `varchar(100)` | NOT NULL, UNIQUE per flow | Stable string identifier (used in routing logic and URL interpolation) | -| `Type` | `integer` | NOT NULL | Enum: `Form=0`, `Information=1`, `Logic=2`, `Redirect=3`, `DocumentUpload=4` | +| `Type` | `integer` | NOT NULL | Enum: `Form=0`, `DocumentUpload=1`, `Redirect=2`, `Information=3`, `Logic=4`. Stored as the ordinal with no value conversion, so these integers are what is in the column. | | `Title` | `varchar(200)` | NOT NULL | Display title shown to the end user | | `JsonContent` | `text` | NOT NULL | Type-specific configuration JSON (field definitions, upload constraints, redirect URL, etc.) | | `ComplianceRuleJson` | `text` | nullable | Server-side validation rules (see [ComplianceRuleJson reference](#compliancerulejson-reference)) | @@ -246,6 +253,58 @@ Immutable snapshots of a flow graph captured on each publish. Enables rollback a **Indexes:** `(FlowId, VersionNumber)` UNIQUE +#### `AnalyticsEvents` + +The durable journey event trail. Added so the per-step record of where applicants stall could be +queried and exported rather than only written to the application log. Read per session by +`GET /api/analytics/sessions/{sessionId}/events`, and surfaced in the operator console under a +session's **Event trail**. + +| Column | Type | Constraints | Description | +|--------|------|-------------|-------------| +| `Id` | `uuid` | PK | The originating event's id; doubles as the idempotency key for client retries | +| `EventType` | `varchar(100)` | NOT NULL | For example `session_started`, `step_view`, `journey_complete` | +| `JourneyId` | `varchar(100)` | NOT NULL | The flow the event belongs to | +| `SessionId` | `varchar(100)` | NOT NULL | The session the event belongs to. A string, not an FK — events are pruned on their own schedule and must not depend on the session row surviving | +| `StepId` | `varchar(100)` | nullable | Node the event relates to, when there is one | +| `StepIndex` | `integer` | nullable | Zero-based position within the session's submission history | +| `PayloadJson` | `text` | NOT NULL | Event-specific payload, so a new event type needs no schema change | +| `Source` | `varchar(20)` | NOT NULL | `server` or `client`. Stamped by the API, never taken from the caller, so a client cannot pass its events off as server-raised | +| `OccurredAt` | `timestamptz` | NOT NULL | When the event happened, as reported by its origin | +| `RecordedAt` | `timestamptz` | NOT NULL | When the row was written. Retention is measured from this, not `OccurredAt`, which a client controls and could backdate | + +**Indexes:** `(SessionId, OccurredAt)` — reading one session's trail in order is the common query + +Pruned by a background sweep after `Analytics__RetentionDays` (default 365). Set +`Analytics__DatabaseProvider__Enabled=false` to disable storage entirely; the application keeps +working and events go only to the log. + +--- + +#### `SessionReadModels` + +A denormalised projection of a session, maintained so the operator session list does not join five +tables per row. Not a source of truth: every column is derived from `Sessions` and its relations. + +| Column | Type | Constraints | Description | +|--------|------|-------------|-------------| +| `Id` | `uuid` | PK | Mirrors `Sessions.Id` | +| `FlowId` | `uuid` | NOT NULL | The flow being executed | +| `FlowName` | `text` | NOT NULL | Denormalised from `Flows.Name` | +| `CustomerEmail` | `text` | nullable | Denormalised from the linked `CustomerProfiles` row | +| `CustomerCountry` | `text` | nullable | Denormalised from the linked `CustomerProfiles` row | +| `ExternalCustomerId` | `text` | nullable | Denormalised from the linked `CustomerProfiles` row | +| `CurrentNodeId` | `uuid` | nullable | Node awaiting submission; `null` once terminal | +| `CurrentNodeKey` | `text` | nullable | Denormalised from `Nodes.Key` | +| `CurrentNodeTitle` | `text` | nullable | Denormalised from `Nodes.Title` | +| `StatusName` | `text` | NOT NULL | Status as a string, so the list needs no enum mapping | +| `StepCount` | `integer` | NOT NULL | Number of submissions recorded | +| `CreatedAt` | `timestamptz` | NOT NULL | Session creation timestamp | +| `UpdatedAt` | `timestamptz` | NOT NULL | Last modification timestamp | +| `CompletedAt` | `timestamptz` | nullable | Set when the session completes | + +--- + ### Design notes **All primary keys are UUIDs** — generated by the caller for flows and nodes (to support offline graph construction in the builder), generated server-side for runtime records (sessions, submissions, deliveries). diff --git a/docs/journey-json.md b/docs/journey-json.md index eb6c65d..c395244 100644 --- a/docs/journey-json.md +++ b/docs/journey-json.md @@ -124,7 +124,7 @@ The journey JSON links directly to frontend behavior: | `connections[]` | Runtime engine + `JourneyBuilder.tsx` | Drives routing and shows labeled graph edges in the read-only session monitor. | | `nodes[]`/`connections[]` | `VisualJourneyBuilder.tsx` + `VisualJourneyCanvas.tsx` | Edited visually via drag-and-drop in the admin builder at `/admin/journey-builder`. | | `nodes[]`/`connections[]` | `FlowAuthoringPanel.tsx` + `flowAuthoring.ts` | Alternative JSON authoring panel for power users; runs alongside the visual builder. | -| `nodes[].type` | `VisualJourneyCanvas.tsx` (`NODE_TYPE_STYLES`) | Determines the color scheme of each node on the canvas (Form=blue, DocumentUpload=purple, Redirect=amber, Information=green, Logic=orange). | +| `nodes[].type` | `visualJourneyCanvasUtils.ts` (`NODE_TYPE_STYLES`) | Determines the color scheme of each node on the canvas (Form=blue, DocumentUpload=purple, Redirect=amber, Information=green, Logic=orange). | | `nodes[].isStartNode` | `VisualJourneyCanvas.tsx` + `NodePropertiesPanel.tsx` | Start node receives a bold outline; Properties Panel enforces at most one start node. | ## Journey visualization diff --git a/docs/runbook.md b/docs/runbook.md index 280583f..24ba5f2 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -295,21 +295,41 @@ Docker Compose configures automatic restart when the liveness probe fails. The API exposes Prometheus metrics at `GET /metrics`. -Key metrics to monitor: - -| Metric | Description | Alert threshold | -|--------|-------------|-----------------| -| `dotnet_duration_seconds` | HTTP request duration | p95 > 2s | -| `waymark_session_started_total` | New sessions started | Sudden drop | -| `waymark_session_completed_total` | Sessions completed | Sudden drop | -| `waymark_session_abandoned_total` | Sessions abandoned by timeout | Spike > normal | -| `waymark_webhook_delivery_failed_total` | Failed webhook deliveries | > 5 in 5 min | -| `waymark_webhook_delivery_retried_total` | Webhook retries | Sustained high rate | -| `process_cpu_seconds_total` | Process CPU usage | > 80% for 5 min | - -**Grafana dashboard** (if using Grafana Cloud / self-hosted): -- Import the provided dashboard JSON from `docs/grafana-dashboard.json` (if available) -- Data source: Prometheus scraping `http://api:8080/metrics` every 15s +The endpoint requires authentication and is exempt from rate limiting. + +### Application metrics + +These are the metrics the application emits. Names are as registered in +`PrometheusMetricsService`; build alerts against these exact strings. + +| Metric | Type | Labels | Emitted when | Alert on | +|--------|------|--------|--------------|----------| +| `onboarding_sessions_started_total` | counter | `flowId` | A session starts | Sudden drop | +| `onboarding_sessions_completed_total` | counter | `flowId` | A session completes | Sudden drop, or a widening gap against started | +| `onboarding_webhook_deliveries_total` | counter | `status` (`delivered`, `failed`) | Each delivery attempt resolves | `failed` rate > 5 in 5 min | +| `waymark_virus_scan_bypassed_total` | counter | — | A document upload is accepted without a real scan | Any increase outside a deliberately unscanned environment | + +`prometheus-net` also exports standard .NET runtime and process metrics on the same endpoint, among +them `process_cpu_seconds_total` (alert above 80% for 5 min) and the GC and thread-pool series. + +### What is not exposed + +Worth knowing before you design a dashboard around metrics that will never arrive: + +- **HTTP request duration and status code counts.** `UseHttpMetrics()` is not wired up, so there is + no `http_request_duration_seconds` series and no per-endpoint latency or error rate. Request-level + monitoring has to come from your ingress or APM until that middleware is added. +- **Abandoned sessions.** There is no counter for sessions the timeout sweep abandons. Track + abandonment from the `Sessions` table or the analytics event trail instead. +- **`onboarding_active_sessions`.** The gauge is registered but nothing ever sets it, so it reads + zero permanently. Do not alert on it. + +### Grafana + +There is no dashboard JSON in this repository; build one against the metric names above. + +- Data source: Prometheus scraping `http://api:8080/metrics` every 15s. +- The scraper needs a credential, since `/metrics` requires authentication. --- diff --git a/website/docs/test-journeys.md b/website/docs/test-journeys.md index b9b2f85..b3645e0 100644 --- a/website/docs/test-journeys.md +++ b/website/docs/test-journeys.md @@ -28,21 +28,21 @@ Each journey isolates a distinct capability of the platform. ```bash # 1. Start session curl -X POST http://localhost:5072/api/workflow/sessions/start \ - -H 'X-Api-Key: dev-key' \ + -H 'X-Api-Key: dev-api-key-change-in-production' \ -H 'Content-Type: application/json' \ -d '{"flowId":"a1a1a1a1-a1a1-a1a1-a1a1-a1a1a1a1a1a1"}' # 2. Submit form (valid) curl -X POST http://localhost:5072/api/workflow/sessions/{sessionId}/steps/{nodeId}/submit \ - -H 'X-Api-Key: dev-key' \ + -H 'X-Api-Key: dev-api-key-change-in-production' \ -H 'Content-Type: application/json' \ - -d '{"FullName":"Alice Example","Email":"alice@example.com"}' + -d '{"payload":{"FullName":"Alice Example","Email":"alice@example.com"}}' # 3. Submit Information node (empty payload advances session) curl -X POST http://localhost:5072/api/workflow/sessions/{sessionId}/steps/{nodeId}/submit \ - -H 'X-Api-Key: dev-key' \ + -H 'X-Api-Key: dev-api-key-change-in-production' \ -H 'Content-Type: application/json' \ - -d '{}' + -d '{"payload":{}}' # → {"isCompleted":true,"currentNode":null} ``` @@ -77,11 +77,11 @@ See [`src/frontend/src/schemas/journey-a-linear-basic.json`](../../src/frontend/ ```bash # France → EU disclosure -curl -X POST .../steps/{nodeId}/submit -d '{"Country":"France"}' +curl -X POST .../steps/{nodeId}/submit -d '{"payload":{"Country":"France"}}' # → currentNode.key = "eu-gdpr-disclosure" # USA → global terms -curl -X POST .../steps/{nodeId}/submit -d '{"Country":"USA"}' +curl -X POST .../steps/{nodeId}/submit -d '{"payload":{"Country":"USA"}}' # → currentNode.key = "global-terms-and-conditions" ``` @@ -126,15 +126,15 @@ The raw `{{sessionId}}` token is **never** sent to the frontend. curl -X POST .../sessions/start -d '{"flowId":"c3c3c3c3-..."}' # 2a. Invalid NationalId → 422 -curl -X POST .../steps/{nodeId}/submit -d '{"NationalId":"INVALID"}' +curl -X POST .../steps/{nodeId}/submit -d '{"payload":{"NationalId":"INVALID"}}' # → 422 {"violations":[{"field":"NationalId","message":"..."}]} # 2b. Valid NationalId → advances to DocumentUpload -curl -X POST .../steps/{nodeId}/submit -d '{"NationalId":"AB123456"}' +curl -X POST .../steps/{nodeId}/submit -d '{"payload":{"NationalId":"AB123456"}}' # → currentNode.type = "DocumentUpload" # 3. DocumentUpload submission (empty payload in test mode skips virus scan) -curl -X POST .../steps/{nodeId}/submit -d '{}' +curl -X POST .../steps/{nodeId}/submit -d '{"payload":{}}' # → currentNode.type = "Redirect", jsonContent.url contains real sessionId ``` diff --git a/website/index.html b/website/index.html index b89906f..8f46ced 100644 --- a/website/index.html +++ b/website/index.html @@ -57,7 +57,7 @@

Waymark decouples journey logic from your frontend and backend code. - Define states, transitions, and conditions in structured JSON or YAML — + Define states, transitions, and conditions as structured JSON — and let the engine handle the rest.

@@ -319,7 +319,7 @@

Everything you need t

Schema-First Design

-

Define states, transitions, and conditions in JSON or YAML. The engine interprets your schema at runtime — no compilation required.

+

Define states, transitions, and conditions as JSON. The engine interprets your schema at runtime — no compilation required.

@@ -351,7 +351,7 @@

Document Upload

-

Built-in DocumentUpload node type with file type filtering, size limits, and progress streaming via SSE.

+

Built-in DocumentUpload node type with file type filtering, size limits, per-file upload progress, and optional virus scanning.