Enforce every grammar key in every consumer

CONTRIBUTING.md

@@ -1,83 +1,183 @@
 # Contributing to Stackform
 
-Thanks for your interest in the project. This guide covers the development setup, the test suite,
-and what a reviewable change looks like.
+Thank you for your interest in Stackform. Bug reports, documentation fixes and code changes are all
+welcome. This guide explains how to get in touch, how to set up a development environment, and what
+a pull request needs before it can be merged.
+
+For what Stackform does and how to use it, read the [README](README.md) and the
+[manual](https://sebastiaoteixeira.github.io/yaml-framework-docs/).
+
+## Questions, bugs and ideas
+
+All conversation happens in [GitHub issues](https://github.com/ieeta-pt/stackform/issues). Search
+the open and closed issues first; your question or bug may already have an answer.
+
+- **Questions.** Open an issue and describe what you are trying to do.
+- **Bug reports.** Include:
+  - the Stackform commit or version and your Python version;
+  - the smallest YAML spec that reproduces the problem, including its `version:` line;
+  - the exact command you ran (for example `stackform build spec.yaml --target drf`);
+  - what you expected and what happened, with the full error output.
+- **Feature proposals.** Describe the problem first, then the change you suggest. For a change to
+  the specification language, show the YAML a user would write.
+
+## Before you write code
+
+Small fixes such as typos, broken links or an obvious one-line bug can go straight to a pull
+request. For anything larger, open an issue first and wait for a maintainer to agree on the
+approach. This applies in particular to new spec keys or changes to existing ones, which every
+backend target has to support. Issues labelled
+[`good first issue`](https://github.com/ieeta-pt/stackform/labels/good%20first%20issue) or
+[`help wanted`](https://github.com/ieeta-pt/stackform/labels/help%20wanted) are open for anyone to
+take; leave a comment saying you are working on one.
 
 ## Development setup
 
-The project is managed with [uv](https://docs.astral.sh/uv/). A locked environment for the
-supported interpreters is created with:
+You need:
+
+- Python 3.11 or newer (CI tests 3.11 and 3.13);
+- [uv](https://docs.astral.sh/uv/), which installs the locked dependencies;
+- Docker, for the integration tests;
+- Node.js 18 or newer, only for frontend generation and the tests that run generated frontend code.
+
+Fork the repository, then clone your fork and install the project with its development
+dependencies:
 
 ```bash
-uv sync --locked --all-extras
+git clone https://github.com/<your-username>/stackform.git
+cd stackform
+uv sync --all-extras
 ```
 
-A plain virtual environment works too:
+The `stackform` command is now available through `uv run`:
 
 ```bash
-pip install -e ".[dev]"
+uv run stackform --help
 ```
 
-Python 3.11 or newer is required. The integration suite additionally needs a MongoDB instance:
+## Running the tests
+
+The suite has two parts. Tests marked `integration` start real services; everything else runs
+without them.
+
+Run the unit suite, as CI does:
 
 ```bash
-docker run -d --name stackform-mongodb -p 27017:27017 mongo:8
+uv run pytest -m "not integration"
 ```
 
-## Running the tests
+Run the integration suite:
 
 ```bash
-uv run pytest -m "not integration"   # unit suite, no services required
-uv run pytest -m integration         # integration suite, needs MongoDB
-uv run pytest                        # everything
+uv run pytest -m integration
 ```
 
-GitHub Actions runs the unit suite on Python 3.11 and 3.13 for every push to `main` and every pull
-request. A pull request whose unit suite is red is not reviewed.
+Each integration test starts a throwaway Docker container for the service it needs (`mongo:7`,
+`postgres:16` or `minio/minio`). To use a service you already run, set its environment variable
+instead:
+
+| Service    | Variable                     |
+| ---------- | ---------------------------- |
+| MongoDB    | `YAML_API_TEST_MONGO_URI`    |
+| PostgreSQL | `YAML_API_TEST_POSTGRES_URI` |
+| S3 (MinIO) | `YAML_API_TEST_S3_ENDPOINT`  |
+
+Some tests are skipped instead of failing when a tool is missing:
+
+- without Docker and without the variable above, the tests that need that service;
+- without `node` or `npm` on the `PATH`, the tests that run generated frontend code;
+- without `YAML_API_TEST_DRF_PYTHON`, the tests that serve a generated Django REST Framework
+  backend. Set it to a Python interpreter that has the generated DRF project's requirements
+  installed.
 
-## Where the code lives
+CI runs only the unit suite. If your change touches generated backends, datastores or
+authentication, run the integration suite locally as well.
 
-| Path | Responsibility |
-| --- | --- |
-| `src/yaml_api_generator/parsers/` | JSON Schema validation, desugaring, and version transformers that read a spec into the IR |
-| `src/yaml_api_generator/ir/` | the intermediate representation every generator consumes |
-| `src/yaml_api_generator/generators/` | backend, frontend and deploy emitters |
-| `src/yaml_api_generator/server/` | the development server that runs a spec without compiling it |
-| `src/yaml_api_generator/cli/` | the `stackform` command |
-| `examples/` | the specification corpus; `valid/` and `invalid/` are consumed by the tests |
-| `docs/` | the manual, published to the documentation site on merge |
+To run one file or one test, pass it to pytest as usual:
+
+```bash
+uv run pytest tests/unit/test_parser.py -k <name>
+```
+
+## Where things live
+
+| Path                                     | Contents                                                   |
+| ---------------------------------------- | ---------------------------------------------------------- |
+| `src/yaml_api_generator/cli/`            | The `stackform` command                                    |
+| `src/yaml_api_generator/parsers/`        | Spec loading and validation; JSON Schemas in `schemas/`    |
+| `src/yaml_api_generator/ir/`             | The intermediate representation the generators consume     |
+| `src/yaml_api_generator/generators/`     | `backend/` (`fastapi`, `drf`, `raw`), `frontend/`, `deploy/` |
+| `src/yaml_api_generator/server/`         | The development server behind `stackform run`              |
+| `src/yaml_api_generator/runtime/`        | Modules shared by the development server and generated applications |
+| `tests/unit/`, `tests/integration/`      | The two test suites                                        |
+| `tests/fixtures/valid/`, `tests/fixtures/invalid/` | Specs that must load and specs that must be rejected |
+| `examples/`                              | Example specs, grouped by spec version                     |
+| `docs/`                                  | The manual, one directory per spec version                 |
+
+The Python package keeps the name `yaml_api_generator`; the distribution and the command are
+named `stackform`.
 
 ## Changing the specification language
 
-The language is versioned, and each version has its own schema under
-`src/yaml_api_generator/parsers/schemas/`. A change to the language is only complete when all of
-the following land together:
+A spec file selects its grammar with its `version:` key. Each version has a JSON Schema in
+`src/yaml_api_generator/parsers/schemas/` (for example `yaml-spec-v0.3.schema.json`). A pull
+request that adds or changes a spec key carries:
+
+- the schema change and the parser change for the affected version;
+- a spec in `tests/fixtures/valid/` that uses the key, and specs in `tests/fixtures/invalid/` for
+  each way it can be misused, with unit tests that load them;
+- the generator change for every backend target that supports the feature;
+- the manual page for the key in `docs/<version>/`, and an entry in that version's
+  `grammar-reference.md`.
 
-1. the schema for the version that introduces it,
-2. a transformer under `parsers/transformers/` if an older version must keep parsing,
-3. an example under `examples/valid/` (and, where the change adds a rejection, one under
-   `examples/invalid/`),
-4. unit tests covering both the accepted and the rejected shapes,
-5. the manual page under `docs/` for that version.
+## Documentation
 
-A specification feature with no example is treated as undocumented and will be asked for in review.
+The manual is written in Markdown under `docs/`, with one directory per spec version. Each page
+starts with front matter that sets `title` and `description`, and each directory's `meta.json`
+sets the page order. When you add a page, add it to `meta.json`.
+
+The site is built with Fumadocs from the separate
+[yaml-framework-docs](https://github.com/sebastiaoteixeira/yaml-framework-docs) repository. A push
+to `main` that changes `docs/` triggers a rebuild. To preview your changes locally, clone that
+repository next to this one and link the content in:
+
+```bash
+git clone https://github.com/sebastiaoteixeira/yaml-framework-docs.git
+cd yaml-framework-docs
+mkdir -p content
+ln -s "$(pwd)/../stackform/docs" content/docs
+npm install
+npm run dev
+```
+
+The site is then served at <http://localhost:3000/yaml-framework-docs/docs>.
 
 ## Pull requests
 
-- Branch from `main`, named `<type>/<subject>` — for example `fix/login-route-404` or
-  `feat/field-level-scoping`.
-- One concern per pull request. A refactor bundled with a behaviour change is split before review.
-- The title states the change in the imperative, and the body says what a reviewer should check.
-- Keep generated output, local scratch files and client material out of the commit; `git status`
-  should be clean of them before you stage.
+1. Create a branch named `<type>/<short-description>`, where the type is `feat`, `fix`, `docs`,
+   `refactor`, `ci` or `chore`. If the branch implements an issue, put its number after the type,
+   for example `feat/83-pagination`.
+2. Keep each pull request to one change. Unrelated fixes go in separate pull requests.
+3. Add or update tests for the behaviour you change, and update the manual if users will notice
+   the change.
+4. Run the unit suite, and the integration suite where it applies.
+5. Open the pull request against `main`. In the description, say what changes and how you tested
+   it, and link the issue it resolves (for example `Closes #83`).
+
+Pull requests are squash-merged, so the pull request title becomes the commit message on `main`.
+Write the title as a single sentence in the imperative mood that states the change, without a type
+prefix or a trailing period. For example:
+
+- `Reject deploy build contexts that enclose the spec or output directory`
+- `Await pymongo async aggregate and list_indexes cursors`
+- `Add file datastores and file field type`
 
-## Reporting issues
+No formatter or linter is enforced; follow the style of the code around your change.
 
-Open an issue with the specification that reproduces the problem, the command you ran, and the
-observed against the expected output. A minimal spec that still fails is worth more than a large
-one that fails for several reasons at once.
+CI runs the unit suite on Python 3.11 and 3.13 for every pull request. A maintainer reviews the pull
+request once CI passes and may ask for changes before merging.
 
 ## Licence
 
-Stackform is released under the MIT licence. By contributing you agree that your contribution is
-licensed under the same terms.
+Stackform is released under the [MIT License](LICENSE.txt). By submitting a contribution, you agree
+that it is licensed under the same terms.

docs/index.md

@@ -103,5 +103,5 @@ YAML Spec --> Parser --> Intermediate Representation --> Code Generator --> Fast
 
 ### Spec Documents
 
-- [YAML Specification v0.1](./YAML-Specification-v0.1.md)
-- [YAML Structure Spec v0.1](./YAML-Structure-Spec-v0.1.md)
+- [v0.1 Specification](./v0.1/specification)
+- [v0.1 Grammar Reference](./v0.1/grammar-reference)

docs/v0.1/grammar-reference.md

@@ -0,0 +1,156 @@
+---
+title: Grammar Reference
+description: Every key a v0.1 spec accepts, with its type, in one tree.
+---
+
+This tree lists every key a v0.1 spec accepts. Every object is closed: a key
+that does not appear here fails the build with `SCHEMA_VALIDATION_ERROR`
+(`'<key>' was unexpected`). The [specification](./specification.md) describes
+what each key does.
+
+Notation:
+
+- `!` = required
+- `?` = optional
+- `map<K,V>` = mapping from names to values
+- `enum(...)` = one of the listed values
+- `array<T>` = list of values
+- `oneOf(...)` = one of the listed forms
+
+```yaml
+YAML-SPEC:
+  version!: "0.1"
+
+  api?:
+    basePath?: string                         # default: /api
+    version?: string
+    pagination?:
+      defaultPageSize?: integer >= 1          # default: the smaller of 20 and maxPageSize
+      maxPageSize?: integer >= 1              # default: 100
+    operations?:
+      defaults?: OperationFlags
+
+  mongo?:
+    uri?: string                              # default: mongodb://localhost:27017
+    database?: string                         # default: yaml_api
+
+  security?:
+    cors?:
+      allowedOrigins?: array<string>          # no "*" when allowCredentials is true
+      allowCredentials?: boolean              # default: false
+      allowedMethods?: array<string>          # default: ["*"]
+    headers?:
+      enableStandard?: boolean                # default: true
+
+  providers?: map<string, ProviderConfig>
+
+  collections!: map<string, CollectionConfig>
+
+  ui?: UiConfig                               # see the UI reference
+
+
+OperationFlags:
+  list?: boolean
+  get?: boolean
+  create?: boolean
+  update?: boolean
+  replace?: boolean
+  delete?: boolean
+
+
+ProviderConfig: oneOf(OIDCProvider, SAMLProvider)
+
+OIDCProvider:
+  type!: oidc
+  client_id!: string
+  client_secret!: string
+
+SAMLProvider:
+  type!: saml
+  idp_metadata_url!: string
+  sp_entity_id!: string
+
+
+CollectionConfig:
+  type?: enum(base, auth)                     # default: base
+  fields!: map<string, FieldConfig>           # exactly one field of type id
+  auth?: AuthConfig                           # required when type is auth
+  roles?: RolesConfig                         # only when type is auth
+  operations?: OperationFlags
+  access?:
+    list?: Access
+    get?: Access
+    create?: Access
+    update?: Access
+    replace?: Access                          # default: the update access
+    delete?: Access
+  softDelete?: boolean                        # default: false
+
+
+AuthConfig:
+  identifier!: string                         # a field of the collection
+  via!: oneOf(string, array<string>)          # password, apiKey or a providers key
+
+
+RolesConfig:
+  values!: array<string>                      # at least one role
+  default!: string                            # one of values
+  hierarchy?: map<string, array<string>>      # role -> roles it includes
+
+
+FieldConfig:                                  # a top-level collection field
+  <the keys of its type, below>
+  unique?: boolean
+
+NestedFieldConfig:                            # an object property
+  <the keys of its type, below>
+
+ItemConfig:                                   # the items of an array
+  type!: <a field type>
+  <the keys added by its type, below>
+
+Keys of every type, except in ItemConfig:
+  type!: enum(string, number, boolean, datetime, enum, array, object, ref, id)
+  required?: boolean
+  default?: any                               # must satisfy the field's own keys
+
+Keys added by type:
+  string:
+    minLength?: integer >= 0
+    maxLength?: integer >= 0
+    format?: enum(email, e164, uri, uuidv7, uuidv4)
+  number:
+    min?: number
+    max?: number
+  enum:
+    values!: array<string>                    # at least one value
+  array:
+    items!: ItemConfig
+  object:
+    properties?: map<string, NestedFieldConfig>
+    additionalProperties?: boolean            # default: false with properties, true without
+  ref:
+    ref!: string                              # a declared collection
+  id:
+    format?: enum(uuidv7, uuidv4)             # default: uuidv7
+  boolean, datetime: no further keys
+
+
+Access: oneOf(Principal, array<AccessRule>)
+
+Principal: oneOf(
+  public,
+  "*",
+  <authCollection>:*,
+  <authCollection>:<role>,
+  <role>                                      # only with a single auth collection
+)
+
+AccessRule:
+  principal!: Principal
+  when?: string                               # when expression
+  reason?: string                             # 403 detail when the rule refuses
+```
+
+The `ui` block has the same shape in every spec version; see the
+[UI reference](../v0.2/Reference/ui/overview.md).