diff --git a/AGENTS.md b/AGENTS.md index 71d54049..659b8817 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -41,6 +41,9 @@ session links out of tracked files, commit messages, and pull request descriptions. - Keep local agent caches, indexes, evaluation records, and browser evidence out of both Git and the Docker build context. Git ignores do not exclude Docker inputs. +- Design proposals live in `docs/proposals/NNNN-slug.md` and decision records + in `docs/adr/NNNN-slug.md`. Agent working plans and scratch specs stay out of + the repository. ## Control database diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a7aaa9f4..37bd3579 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -133,8 +133,18 @@ cmd/fanout/ process composition and the single entry point internal/ ingest, storage, query, agent, MCP, auth, alerts ui/host/ React browser host and shared MCP app renderer (build-time) docs/diagrams/ archify JSON sources and exported dual-theme SVGs +docs/proposals/ numbered design proposals for significant changes +docs/adr/ numbered architecture decision records ``` +## Design proposals and decisions + +Write a [design proposal](docs/proposals/README.md) before building a +significant change to a capability, a public contract or a cross-cutting +design. Record a settled decision that should not be re-argued as an +[ADR](docs/adr/README.md). Agent working plans and scratch notes stay out of +the repository. + ## Commits and pull requests Write commit messages that explain why the change is needed, not only what it diff --git a/context7.json b/context7.json index 3f8e1df8..9c8e1ff9 100644 --- a/context7.json +++ b/context7.json @@ -5,7 +5,8 @@ "branch": "main", "folders": ["site/src/content/docs", "docs", "README.md"], "excludeFolders": [ - "docs/specs", + "docs/proposals", + "docs/adr", "docs/media", "docs/diagrams" ], diff --git a/docs/adr/0001-environment-only-configuration.md b/docs/adr/0001-environment-only-configuration.md new file mode 100644 index 00000000..39f43462 --- /dev/null +++ b/docs/adr/0001-environment-only-configuration.md @@ -0,0 +1,26 @@ +# 0001. Environment-only configuration + +Status: Accepted +Date: 2026-10-30 + +## Context + +Fanout runs as one binary on laptops, containers and hosted platforms. It +needs one configuration source that works the same way in each, and a way to +deliver secrets without writing them into configuration. + +## Decision + +Configuration comes only from the process environment, over built-in defaults. +`internal/config/config.go` is the authoritative schema. Custom names use the +`FANOUT_` prefix; vendor credentials and endpoints keep their vendor names +(`ANTHROPIC_API_KEY`, `OPENAI_BASE_URL`, …). Every secret also accepts a +`_FILE` form. Unknown `FANOUT_` names fail startup. The binary does not +load dotenv files. + +## Consequences + +- One precedence rule and one place to read the schema. +- `.env.example` and the site reference are generated from the schema. +- Secrets can be mounted as files without appearing in the environment. +- A typo in a `FANOUT_` name fails fast instead of being ignored. diff --git a/docs/adr/0002-sqlite-goose-sqlc-control-state.md b/docs/adr/0002-sqlite-goose-sqlc-control-state.md new file mode 100644 index 00000000..43939adf --- /dev/null +++ b/docs/adr/0002-sqlite-goose-sqlc-control-state.md @@ -0,0 +1,25 @@ +# 0002. SQLite with Goose and sqlc for control state + +Status: Accepted +Date: 2026-10-30 + +## Context + +Fanout needs a transactional store for users, sessions, dashboards, alerts and +settings, without adding a database server to operate. + +## Decision + +Control state lives in SQLite through `modernc.org/sqlite` and `database/sql`, +with sqlc-generated query bindings and timestamped Goose SQL migrations. +`internal/db/migrations` is the single schema source for both Goose and sqlc. +Published migrations are immutable and checksummed in +`internal/db/migrations_test.go`. PostgreSQL and other migration frameworks are +not used for control state. + +## Consequences + +- No external database to deploy. +- Schema changes always add a new migration and run `just db-gen`. +- A database with application tables must have a positive applied Goose + version before initialization; there is no schema fallback. diff --git a/docs/adr/0003-pinned-duckdb-and-batch-format-4.md b/docs/adr/0003-pinned-duckdb-and-batch-format-4.md new file mode 100644 index 00000000..ebe4d841 --- /dev/null +++ b/docs/adr/0003-pinned-duckdb-and-batch-format-4.md @@ -0,0 +1,26 @@ +# 0003. Pinned DuckDB engine and batch format 4 + +Status: Accepted +Date: 2026-10-30 + +## Context + +Telemetry queries depend on exact engine behaviour: built-in functions, Parquet +VARIANT handling and nanosecond timestamps. A Go driver version does not +identify the native engine it bundles, and silent engine or format drift would +change query results. + +## Decision + +Fanout builds against one pinned DuckDB 2 engine (`internal/duckdb`, built with +`scripts/with-duckdb.sh`) with statically linked extensions and no dynamic +extension downloads. Telemetry is stored only as batch format 4 Parquet with +typed attribute maps. A schema change requires a new format version; there is +no legacy reader or format fallback. + +## Consequences + +- Query results are reproducible across platforms and releases. +- Changing the pin means verifying `SELECT version()` and reviewing every + changed built-in against the telemetry SQL blocklist. +- Data written in an unsupported format is rejected rather than reinterpreted. diff --git a/docs/adr/0004-no-legacy-paths.md b/docs/adr/0004-no-legacy-paths.md new file mode 100644 index 00000000..8f094cdf --- /dev/null +++ b/docs/adr/0004-no-legacy-paths.md @@ -0,0 +1,26 @@ +# 0004. No legacy paths + +Status: Accepted +Date: 2026-10-30 + +## Context + +Keeping aliases, fallback routes, old readers or schema fallbacks next to their +replacements doubles the code to test and lets superseded behaviour linger. + +## Decision + +Superseded behaviour is removed in the same change that replaces it: + +- Breaking naming changes update registrations, authorization classification, + clients, tests, metrics configuration and generated documentation together, + and remove superseded routes and handlers with no aliases or fallback. +- The control database has no legacy migration tracker or schema fallback. +- Telemetry storage accepts only the current batch format, with no legacy + reader or format fallback. + +## Consequences + +- One code path per behaviour, and less to review. +- A breaking change can require updating clients, or resetting data written in + an unsupported format. diff --git a/docs/adr/0005-calver-releases.md b/docs/adr/0005-calver-releases.md new file mode 100644 index 00000000..e5415b76 --- /dev/null +++ b/docs/adr/0005-calver-releases.md @@ -0,0 +1,22 @@ +# 0005. CalVer releases + +Status: Accepted +Date: 2026-10-30 + +## Context + +Fanout ships frequently and its maturity is better described by date and stage +than by semantic version numbers. + +## Decision + +Releases use CalVer `YYYY.M.N[-alpha|-beta|-rc]`: full year, unpadded month, +and a release number within the month that starts at 0 and increments for +every published release. The suffix states maturity; GA has none. Git tags +carry a leading `v`; container tags do not. Published versions are immutable. + +## Consequences + +- A version tells when it was built and how mature it is. +- Product versions, client-contract versions and protocol versions are kept + separate; the product version never appears in API paths or tool names. diff --git a/docs/adr/0006-one-structured-query-path.md b/docs/adr/0006-one-structured-query-path.md new file mode 100644 index 00000000..02682f8b --- /dev/null +++ b/docs/adr/0006-one-structured-query-path.md @@ -0,0 +1,26 @@ +# 0006. One structured query path + +Status: Accepted +Date: 2026-10-30 + +## Context + +Telemetry is queried from dashboards, FanQL text, MCP tools and the agent, and +a visual builder is planned. Separate query paths would each need their own validation, +limits and SQL generation, and could diverge. + +## Decision + +Every surface produces the same validated structured query (`from`, `where`, +`measures`, `by`, `bucket`, `histogram`, `sort`, `limit`), which one engine path +validates, budgets and turns into SQL. FanQL is the text form of that query and +compiles to it; its filters become predicates inside the query, checked by the +same validation, and it never runs SQL of its own. Arbitrary read-only SQL remains a +separate, locked-down gate for what the structured query cannot express. + +## Consequences + +- One place enforces relations, functions and budgets. +- A FanQL query executes through the same engine path as the equivalent + dashboard panel. +- New query features are added once and become available everywhere. diff --git a/docs/adr/0007-hand-written-fanql-parser.md b/docs/adr/0007-hand-written-fanql-parser.md new file mode 100644 index 00000000..54d76f10 --- /dev/null +++ b/docs/adr/0007-hand-written-fanql-parser.md @@ -0,0 +1,23 @@ +# 0007. A hand-written FanQL parser + +Status: Accepted +Date: 2026-10-30 + +## Context + +FanQL needs precise, recovering diagnostics with source spans and suggested +fixes, position-aware completion, and canonical printing, all fast enough to +run on every keystroke. + +## Decision + +FanQL uses a hand-written lexer and recursive-descent parser in Go, with no +parser-generator or grammar library. It recovers after errors so it can report +every independent problem in one pass. + +## Consequences + +- Diagnostics, completion and printing share one parser and one tree. +- No generated code or external grammar toolchain to maintain. +- Grammar changes are code changes, covered by the corpus, round-trip and fuzz + tests. diff --git a/docs/adr/README.md b/docs/adr/README.md new file mode 100644 index 00000000..27202757 --- /dev/null +++ b/docs/adr/README.md @@ -0,0 +1,34 @@ +# Architecture decision records + +An ADR records one decision that shapes the code, with its context and +consequences, in a page or less. [AGENTS.md](../../AGENTS.md) holds the binding +rules; an ADR records why a rule exists. Write one when a decision would +otherwise be re-argued. Supersede an ADR with a new one rather than editing its +decision. + +ADRs are numbered in order: `NNNN-short-slug.md`. + +## Template + +```markdown +# NNNN. Title + +Status: Accepted | Superseded by NNNN +Date: YYYY-MM-DD + +## Context +## Decision +## Consequences +``` + +## Index + +| # | Decision | Status | +|---|---|---| +| 0001 | [Environment-only configuration](0001-environment-only-configuration.md) | Accepted | +| 0002 | [SQLite with Goose and sqlc for control state](0002-sqlite-goose-sqlc-control-state.md) | Accepted | +| 0003 | [Pinned DuckDB engine and batch format 4](0003-pinned-duckdb-and-batch-format-4.md) | Accepted | +| 0004 | [No legacy paths](0004-no-legacy-paths.md) | Accepted | +| 0005 | [CalVer releases](0005-calver-releases.md) | Accepted | +| 0006 | [One structured query path](0006-one-structured-query-path.md) | Accepted | +| 0007 | [A hand-written FanQL parser](0007-hand-written-fanql-parser.md) | Accepted | diff --git a/docs/specs/2026-09-05-logged-in-ui-shell-design.md b/docs/proposals/0001-logged-in-ui-shell.md similarity index 99% rename from docs/specs/2026-09-05-logged-in-ui-shell-design.md rename to docs/proposals/0001-logged-in-ui-shell.md index b4a042f6..a09f6bda 100644 --- a/docs/specs/2026-09-05-logged-in-ui-shell-design.md +++ b/docs/proposals/0001-logged-in-ui-shell.md @@ -1,7 +1,8 @@ # Logged-in UI: shell rebuild, chat pane, native dashboard widgets, sign-in code step +Status: Implemented Date: 2026-09-05 -Status: approved design, pending implementation plan +Pull request: #226 Scope: `ui/host` (React host app), `ui/apps` (embedded MCP views, small changes), `ui/` shared modules. No Go changes are expected; the server widget allowlist and routes stay as they are. ## Why diff --git a/docs/specs/2026-10-04-agent-dashboards-design.md b/docs/proposals/0002-agent-dashboards.md similarity index 99% rename from docs/specs/2026-10-04-agent-dashboards-design.md rename to docs/proposals/0002-agent-dashboards.md index ef5f0bd9..80cc1075 100644 --- a/docs/specs/2026-10-04-agent-dashboards-design.md +++ b/docs/proposals/0002-agent-dashboards.md @@ -1,6 +1,8 @@ # Agent-built dashboards -Status: draft for review · 2026-10-04 · branch `feat/agent-dashboards` +Status: Implemented +Date: 2026-10-04 +Pull requests: #285, #286, #287, #295 (milestones 1 to 4) ## Goal diff --git a/docs/specs/2026-10-10-agent-dashboards-m5-design.md b/docs/proposals/0003-agent-dashboards-m5.md similarity index 99% rename from docs/specs/2026-10-10-agent-dashboards-m5-design.md rename to docs/proposals/0003-agent-dashboards-m5.md index c884c259..42a3a3fa 100644 --- a/docs/specs/2026-10-10-agent-dashboards-m5-design.md +++ b/docs/proposals/0003-agent-dashboards-m5.md @@ -1,8 +1,10 @@ # Milestone 5: correct metrics and the system page -Status: draft for review · 2026-10-10 · branch `feat/agent-dashboards-m5` +Status: Implemented +Date: 2026-10-10 +Pull request: #297 -This extends `2026-10-04-agent-dashboards-design.md`. Milestones 1 to 4 are +This extends [0002-agent-dashboards.md](0002-agent-dashboards.md). Milestones 1 to 4 are merged. Its success criteria S1 to S13 stay in force; this milestone adds S14 to S16, S20 and S22. Performance work (a scale benchmark, per-minute summaries and metric query speed) is deferred to a later milestone. diff --git a/docs/specs/2026-10-29-no-llm-ui-design.md b/docs/proposals/0004-fanql.md similarity index 99% rename from docs/specs/2026-10-29-no-llm-ui-design.md rename to docs/proposals/0004-fanql.md index 882b8e16..61c3edf4 100644 --- a/docs/specs/2026-10-29-no-llm-ui-design.md +++ b/docs/proposals/0004-fanql.md @@ -1,6 +1,8 @@ # Full UI without an LLM key -Design, 2026-10-29. Status: draft for review; section 0 (FanQL) is owner-approved as of 2026-10-29. +Status: Accepted +Date: 2026-10-29 +Pull request: #410 (section 0, FanQL); sections 1 to 5 are not yet built ## Intent diff --git a/docs/proposals/README.md b/docs/proposals/README.md new file mode 100644 index 00000000..33cef7ab --- /dev/null +++ b/docs/proposals/README.md @@ -0,0 +1,40 @@ +# Design proposals + +A proposal describes a significant change before it is built: what it does, +why, and what was considered instead. Write one when a change adds or changes +a product capability, a public contract (HTTP, MCP, configuration, storage +format), or a cross-cutting design. Small fixes and refactors do not need one. + +Proposals are numbered in order: `NNNN-short-slug.md`. Keep the number and file +name when the status changes, so links stay stable. + +## Template + +```markdown +# Title + +Status: Draft | Accepted | Implemented | Superseded by NNNN +Date: YYYY-MM-DD +Pull request: #NNN + +## Summary +## Motivation +## Goals and non-goals +## Design +## Alternatives +## Risks +## Open questions +``` + +Earlier proposals predate the template and keep their original structure. + +## Index + +| # | Proposal | Status | +|---|---|---| +| 0001 | [Logged-in UI: shell, chat pane, native widgets, sign-in code step](0001-logged-in-ui-shell.md) | Implemented | +| 0002 | [Agent-built dashboards](0002-agent-dashboards.md) | Implemented | +| 0003 | [Milestone 5: correct metrics and the system page](0003-agent-dashboards-m5.md) | Implemented | +| 0004 | [Full UI without an LLM key, and FanQL](0004-fanql.md) | Accepted | + +Settled decisions that a proposal leads to are recorded in [../adr](../adr/README.md). diff --git a/internal/fanql/latency_test.go b/internal/fanql/latency_test.go index b2c25d5b..f58082fa 100644 --- a/internal/fanql/latency_test.go +++ b/internal/fanql/latency_test.go @@ -35,7 +35,7 @@ func latencyPercentile(samples []time.Duration, percentile float64) time.Duratio return ordered[max(0, int(math.Ceil(float64(len(ordered))*percentile))-1)] } -// Latency requirements: ../../docs/specs/2026-10-29-no-llm-ui-design.md, +// Latency requirements: ../../docs/proposals/0004-fanql.md, // section 0 (especially 0.14 and 0.16). Tests bound work deterministically; // benchmarks report the strict spec percentiles on an uninstrumented build. func TestLatencyBudgets(t *testing.T) {