Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ The example section has one shape on every provider site, so the table of conten
- Plain prose, `-` rather than em dashes, no ASCII arrows; SQL with uppercase keywords, single-quoted strings, semicolon terminated; placeholders in the same style the generated resource pages use.
- Favicons live at the `static/` root (`favicon.ico`, `favicon-16x16.png`, `favicon-32x32.png`, `apple-touch-icon.png`, `safari-pinned-tab.svg`, `site.webmanifest`); the shared config links them root-relative, so copies under `static/img/` are never served as the site icon.

`make docs` = `generate-docs --snake-case-aliases` + `sanitize-docs.mjs` (escapes MDX-hostile description text, annotates env-var-defaulted parameters "required unless X is set", corrects docgen's landing-page resource count which includes each service index). `make website` = `yarn install && yarn build`. Commit `website/docs` after every regeneration so pages stamp with their regeneration date.
`make docs` = `generate-docs --snake-case-aliases --source-project $(SOURCE_PROJECT)` + `sanitize-docs.mjs` (escapes MDX-hostile description text, annotates env-var-defaulted parameters "required unless X is set", corrects docgen's landing-page resource count which includes each service index). `--source-project` (provider-utils >= 0.7.11) adds a `source project` row to the Provider Summary admonition on the landing page, linking the repository name (the URL's last path segment) to the URL; `sanitize-docs` leaves the row alone. The Makefile carries the URL as `SOURCE_PROJECT ?= https://github.com/stackql-registry/stackql-provider-<slug>` (rewritten by `bin/init-provider.sh`), always the consuming repository's own URL, never the template's; a repository hosted elsewhere sets it there. `make website` = `yarn install && yarn build`. Commit `website/docs` after every regeneration so pages stamp with their regeneration date.

## CI

`build-and-test.yml`: on push/PR - `actions/setup-node@v7` on Node 22, `npm ci`, `stackql/setup-stackql@v2`, `make fetch-spec` (warn on drift, build from the pin), build steps, **fail on uncommitted generation drift** (`git add -N . && git diff --quiet -- provider-dev/openapi provider-dev/config provider-dev/source`), `make test-offline test-integration test-meta`, `make docs`. A `smoke` job gated on secrets (skipped with a notice otherwise; never the expensive lifecycle). A weekly `spec-drift` job that fetches, compares with the pin, and opens a labelled issue with the diff stat. Web deploy workflows build the site from `main` with `yarn install --frozen-lockfile`, cached on `website/yarn.lock`; the first `make website` writes that lockfile, so commit it before enabling them.
`build-and-test.yml`: on push/PR - `actions/setup-node@v7` on Node 22, `npm ci`, `stackql/setup-stackql@v2`, `make fetch-spec` (warn on drift, build from the pin), build steps, **fail on uncommitted generation drift** (`git add -N . && git diff --quiet -- provider-dev/openapi provider-dev/config provider-dev/source`), `make test-offline test-integration test-meta`, `make docs`. A `smoke` job skipped at the job level (no runner consumed) on pull requests, on the schedule, until a spec is pinned and until the secrets are configured: build-and-test exports `pinned` and `smoke_creds` outputs because a job-level `if` cannot read secrets, and raises a notice naming the missing secret where the smoke job would otherwise have run; never the expensive lifecycle. A weekly `spec-drift` job that fetches, compares with the pin, and opens a labelled issue with the diff stat. Web deploy workflows build the site from `main` with `yarn install --frozen-lockfile`, cached on `website/yarn.lock`; the first `make website` writes that lockfile, so commit it before enabling them.
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ website/ # Docusaurus microsite
CLAUDE.md NOTES.md README.md SECURITY.md LICENSE .env.example
```

Makefile targets (bash shell, `.DEFAULT_GOAL := help`, `##` help comments): `deps`, `fetch-spec` (verify pin), `refresh-spec` (`--update`), `inventory`, `split`, `mappings` (rm CSV + analyze + map), `pre-normalize`, `normalize`, `generate` (rm output + generate + post-process), `post-process`, `build` (the chain), `test-offline`, `test-integration`, `test-meta` (start server; run; stop; preserve exit status), `test`, `venv`, `smoke`, `smoke-live`, `smoke-read-only`, `smoke-<gated-lifecycle>`, `smoke-cleanup`, `docs`, `website`, `website-start`, `clean`, `all` = `deps build test docs website`. `make all` never needs credentials and never bills. The smoke targets source `.env` when present (`set -a; source <(tr -d '\r' < .env); set +a`).
Makefile targets (bash shell, `.DEFAULT_GOAL := help`, `##` help comments): `deps`, `fetch-spec` (verify pin), `refresh-spec` (`--update`), `inventory`, `split`, `mappings` (rm CSV + analyze + map), `pre-normalize`, `normalize`, `generate` (rm output + generate + post-process), `post-process`, `build` (the chain), `test-offline`, `test-integration`, `test-meta` (start server; run; stop; preserve exit status), `test`, `venv`, `smoke`, `smoke-live`, `smoke-read-only`, `smoke-<gated-lifecycle>`, `smoke-cleanup`, `docs` (passes `--snake-case-aliases` and `--source-project $(SOURCE_PROJECT)`, the provider repository URL, `?=` so it can be overridden), `website`, `website-start`, `clean`, `all` = `deps build test docs website`. `make all` never needs credentials and never bills. The smoke targets source `.env` when present (`set -a; source <(tr -d '\r' < .env); set +a`).

package.json scripts wrap the two CLI entry points through `node ./node_modules/@stackql/provider-utils/bin/provider-dev-utils.mjs <cmd>` and `docgen-utils.mjs generate-docs` (never `.bin` shims); pass flags after `--`.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Uplifting an existing provider

1. Bump `@stackql/provider-utils` / `@stackql/pgwire-lite` / `@apidevtools/swagger-parser` / Docusaurus to latest (`js-yaml` stays on 4.x - see the SKILL.md ground rules); Node >= 22.19 in `engines`, the Makefile header, the README prerequisites and every `setup-node` step; `npm install`. Add `SECURITY.md` and `LICENSE` from the template when absent.
1. Bump `@stackql/provider-utils` / `@stackql/pgwire-lite` / `@apidevtools/swagger-parser` / Docusaurus to latest (`js-yaml` stays on 4.x - see the SKILL.md ground rules); Node >= 22.19 in `engines`, the Makefile header, the README prerequisites and every `setup-node` step; `npm install`. provider-utils >= 0.7.11: add `SOURCE_PROJECT ?= <the repository URL>` and `--source-project $(SOURCE_PROJECT)` to the `docs` target (docs-and-ci.md). Add `SECURITY.md` and `LICENSE` from the template when absent.
2. `make fetch-spec`; if drift, review the diff, extend fix classes, `make refresh-spec`, record in NOTES.md.
3. Re-run the pipeline; diff `all_services.csv` - every moved method or renamed resource is a decision to make explicitly (keep the old name via a rule unless the rename is worth the break).
4. Add what is missing against this list: env-var scoping, naive bodies, snake surface, pagination, pushdown, transforms, objectKeys, lifecycle EXECs attached to their resources, views, GraphQL merge, skip codes for what is not mappable.
Expand Down
45 changes: 34 additions & 11 deletions .github/workflows/build-and-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,12 @@ jobs:
build-and-test:
if: github.event_name != 'schedule'
runs-on: ubuntu-latest
# Consumed by the smoke job's `if`: a job-level condition cannot read
# secrets or another job's step outputs directly, so both gates are
# computed here and exported.
outputs:
pinned: ${{ steps.pin.outputs.pinned }}
smoke_creds: ${{ steps.creds.outputs.present }}
steps:
- uses: actions/checkout@v7

Expand Down Expand Up @@ -57,6 +63,23 @@ jobs:
echo "::notice title=No pinned spec::provider-dev/config/spec_pin.json does not exist yet (template state) - the build, drift check, tests and docs steps were skipped. Run 'make fetch-spec' to pin the upstream spec."
fi

# Only presence is recorded, never the value. The notice is raised only
# where the smoke job would otherwise have run (a pinned spec, not a
# pull request), so the template and PRs from forks stay quiet.
- name: Detect whether the smoke credentials are configured
id: creds
env:
MYPROVIDER_API_TOKEN: ${{ secrets.MYPROVIDER_API_TOKEN }}
run: |
if [ -n "$MYPROVIDER_API_TOKEN" ]; then
echo "present=true" >> "$GITHUB_OUTPUT"
else
echo "present=false" >> "$GITHUB_OUTPUT"
if [ "${{ steps.pin.outputs.pinned }}" = "true" ] && [ "$GITHUB_EVENT_NAME" != "pull_request" ]; then
echo "::notice title=Live smoke suite skipped::MYPROVIDER_API_TOKEN is not configured - the smoke job was skipped. Credential-free coverage still ran via the mock-server integration suite."
fi
fi

# Drift is reported, not fatal, here: the committed pin is what gets
# built and tested. The spec-drift job below opens the review issue.
- name: Verify the pinned spec against upstream (warns on drift)
Expand Down Expand Up @@ -100,13 +123,19 @@ jobs:
run: make docs

smoke:
# secret-gated live smoke suite against the dedicated dev account;
# skipped with a notice when the secrets are not configured (forks, PRs
# from outside). Never the expensive gated lifecycle - trigger that
# deliberately with `make smoke-gated-lifecycle`.
# Live smoke suite against the dedicated dev account. Skipped at the job
# level, so no runner is consumed, on pull requests (secrets are not
# available to forks), on the schedule, until a spec is pinned, and until
# the secrets are configured; the last two arrive as build-and-test
# outputs. Never the expensive gated lifecycle - trigger that deliberately
# with `make smoke-gated-lifecycle`.
runs-on: ubuntu-latest
needs: build-and-test
if: github.event_name != 'pull_request' && github.event_name != 'schedule'
if: >-
github.event_name != 'pull_request' &&
github.event_name != 'schedule' &&
needs.build-and-test.outputs.pinned == 'true' &&
needs.build-and-test.outputs.smoke_creds == 'true'
env:
MYPROVIDER_API_TOKEN: ${{ secrets.MYPROVIDER_API_TOKEN }}
MYPROVIDER_ORG_ID: ${{ secrets.MYPROVIDER_ORG_ID }}
Expand All @@ -126,14 +155,8 @@ jobs:
run: npm ci

- name: Live smoke suite
if: env.MYPROVIDER_API_TOKEN != ''
run: make smoke

- name: Live smoke skipped (no credentials)
if: env.MYPROVIDER_API_TOKEN == ''
run: |
echo "::notice title=Live smoke suite skipped::MYPROVIDER_API_TOKEN secrets are not configured - the live smoke suite did not run. Credential-free coverage still ran via the mock-server integration suite."

spec-drift:
# Diff the served spec against the pin on a schedule and on demand, and
# open a labelled issue when it moves.
Expand Down
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ TODO(template): replace the paragraph above with what the provider covers (the A

The build procedure is the `stackql-provider-development` skill in `.claude/skills/stackql-provider-development/` (`SKILL.md` is the map; `references/*.md` are the procedures; `scripts/` holds `spec_diff.mjs` and `find_extension_examples.sh`). Read the reference file for a step before executing it. Work the steps in order: archetype -> spec pin -> inventory and split -> mappings -> pre-normalize / normalize / generate / post-process -> GraphQL and views -> Makefile -> the three credential-free test layers -> smoke suite -> docs and CI -> hand-over.

Starting from a fresh clone of the template: `bin/init-provider.sh <name> "<Title>" <api-base>` rewrites the placeholders (`myprovider`, `My Provider`, `MYPROVIDER`, `api.example.com`), then fill the constants in `provider-dev/scripts/lib/spec_helpers.mjs` (`SPEC_URL`, `SPEC_FILE`, `SCOPE_PREFIX`, `ROOT_PATHS`, `WIRE_CASING`) and `provider-dev/config/*.json`, and `make fetch-spec`. Every file carrying a `TODO(template)` marker is a decision the build must make; `grep -rn "TODO(template)" --exclude-dir=node_modules .` lists what is left.
Starting from a fresh clone of the template: `bin/init-provider.sh <name> "<Title>" <api-base>` rewrites the placeholders (`myprovider`, `My Provider`, `MYPROVIDER`, `api.example.com`), then fill the constants in `provider-dev/scripts/lib/spec_helpers.mjs` (`SPEC_URL`, `SPEC_FILE`, `SCOPE_PREFIX`, `ROOT_PATHS`, `WIRE_CASING`) and `provider-dev/config/*.json`, confirm `SOURCE_PROJECT` in the Makefile (the repository URL docgen links from the docs landing page; the default is the `stackql-registry/stackql-provider-<name>` convention), and `make fetch-spec`. Every file carrying a `TODO(template)` marker is a decision the build must make; `grep -rn "TODO(template)" --exclude-dir=node_modules .` lists what is left.

## Archetype

Expand Down Expand Up @@ -82,7 +82,7 @@ Never run tests against a production account.

## Docs and publish

`make docs` (generate with `--snake-case-aliases`, then `sanitize-docs`) and `make website`; `website/docs` is committed after every regeneration so pages stamp with their regeneration date. Publishing to the registry is a separate, human-in-the-loop step (push the generated provider dir to `providers/src` in a feature branch of `stackql-provider-registry`, follow the release flow, verify with `make smoke-live`).
`make docs` (generate with `--snake-case-aliases` and `--source-project $(SOURCE_PROJECT)`, then `sanitize-docs`) and `make website`; `website/docs` is committed after every regeneration so pages stamp with their regeneration date. Publishing to the registry is a separate, human-in-the-loop step (push the generated provider dir to `providers/src` in a feature branch of `stackql-provider-registry`, follow the release flow, verify with `make smoke-live`).

## Writing conventions

Expand Down
13 changes: 10 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# Python 3 (a venv with pystackql is created on demand for the smoke suite),
# yarn for the website. Runs on Linux, macOS or WSL.
#
# TODO(template): set PROVIDER (or run bin/init-provider.sh), then review
# TODO(template): set PROVIDER and SOURCE_PROJECT (or run bin/init-provider.sh), then review
# every target marked TODO as the provider comes together. Rename
# smoke-gated-lifecycle to the expensive lifecycle the provider gates
# (smoke-service, smoke-project-lifecycle, ...), or delete it.
Expand All @@ -30,6 +30,12 @@ PROVIDER_DIR := $(SERVICES_DIR)/$(VERSION)
SOURCE_DIR := provider-dev/source
CONFIG_DIR := provider-dev/config
GRAPHQL_DIR := provider-dev/source-graphql
# The repository docgen links from the Provider Summary on the docs landing
# page (the label is the URL's last path segment). bin/init-provider.sh
# rewrites the slug; a provider hosted outside stackql-registry sets its own
# URL here or passes SOURCE_PROJECT=... on the command line. Always the
# consuming repository's URL, never the template's.
SOURCE_PROJECT ?= https://github.com/stackql-registry/stackql-provider-myprovider
PORT ?= 5444
VENV := .venv
PY := $(VENV)/bin/python
Expand Down Expand Up @@ -156,13 +162,14 @@ smoke-cleanup: venv ## sweep stackql-smoke-* breadcrumbs and exit

# -------------------------------------------------------------------- docs

docs: ## generate the website docs (snake_case surface), then sanitize for MDX
docs: ## generate the website docs (snake_case surface, source project link), then sanitize for MDX
npm run generate-docs -- \
--provider-name $(PROVIDER) \
--provider-dir ./$(PROVIDER_DIR) \
--output-dir ./website \
--provider-data-dir ./provider-dev/docgen/provider-data \
--snake-case-aliases
--snake-case-aliases \
--source-project $(SOURCE_PROJECT)
npm run sanitize-docs

website: ## build the docusaurus microsite (vendors the shared stackql config first)
Expand Down
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ The rest of this file is the shape of the finished provider README (numbered bui
## Using the template

1. Create the repository from this template (GitHub "Use this template", or clone and re-init) and run `npm install`.
2. Rewrite the placeholders: `bin/init-provider.sh <name> "<Title>" [https://api.vendor.com]` replaces `myprovider`, `My Provider`, `MYPROVIDER` and `api.example.com` across the files that carry them (`grep -rn myprovider .` afterwards should find nothing outside `.claude/`).
3. Fill the constants in `provider-dev/scripts/lib/spec_helpers.mjs` (`SPEC_URL`, `SPEC_FILE`, `PATH_VERSION_PREFIX`, `SCOPE_PREFIX` / `ROOT_PATHS` for a scoped API, `WIRE_CASING`) and `provider-dev/config/` (`servers.json`, `provider_config.json`, `service_names.json`).
2. Rewrite the placeholders: `bin/init-provider.sh <name> "<Title>" [https://api.vendor.com]` replaces `myprovider`, `My Provider`, `MYPROVIDER`, `api.example.com` and the `stackql-provider-myprovider` repository slug across the files that carry them (`grep -rn myprovider .` afterwards should find nothing outside `.claude/`).
3. Fill the constants in `provider-dev/scripts/lib/spec_helpers.mjs` (`SPEC_URL`, `SPEC_FILE`, `PATH_VERSION_PREFIX`, `SCOPE_PREFIX` / `ROOT_PATHS` for a scoped API, `WIRE_CASING`) and `provider-dev/config/` (`servers.json`, `provider_config.json`, `service_names.json`). `SOURCE_PROJECT` in the Makefile defaults to `https://github.com/stackql-registry/stackql-provider-<name>`; set it to the repository's own URL when it lives elsewhere.
4. Open a Claude Code session in the repository and ask it to build the provider - it picks up `CLAUDE.md` and the skill. Or work the skill's steps by hand: `make fetch-spec`, `make inventory`, add service rules, `make split`, `make mappings-report` / `make mappings`, and so on to `make all`.
5. `grep -rn "TODO(template)" --exclude-dir=node_modules .` lists what is still open.

Expand Down Expand Up @@ -131,6 +131,8 @@ make website-start

`provider-dev/docgen/provider-data/headerContent1.txt` is the landing-page front matter and pitch; `headerContent2.txt` is the getting-started page (installation, scope, authentication, the scoping variable, rate limit, labelling, example queries - lead with the queries the provider exists for). The examples close the file under an `## Example Queries` heading, the same table-of-contents entry on every provider site: one intro sentence under the H2 (so no heading sits directly on another), then one H3 per example with a one-sentence lead-in ending in a colon and one `sql` block. `website/provider.js` carries the site identity; `website/static/CNAME` the hostname; add `website/static/img/stackql-<name>-provider-featured-image.png`, and keep the favicon files at the `static/` root (the shared config links them root-relative). Commit `website/docs` after every regeneration.

`make docs` also passes `--source-project $(SOURCE_PROJECT)`, which adds a `source project` row to the Provider Summary on the landing page linking to this repository. The default is the `stackql-registry/stackql-provider-<name>` convention; a repository hosted elsewhere sets `SOURCE_PROJECT` in the Makefile (or runs `make docs SOURCE_PROJECT=<url>`).

To publish the site: rename `.github/workflows/prod-web-deploy.yml.disabled` and `test-web-deploy.yml.disabled`, enable GitHub Pages (source: GitHub Actions) and add the DNS record:

| Source domain | Record type | Target |
Expand All @@ -152,7 +154,7 @@ REGISTRY PULL myprovider;

## 8. CI

`.github/workflows/build-and-test.yml`: on push / PR - `npm ci`, `stackql/setup-stackql`, pin verification (warns on drift), the build steps, a hard failure on uncommitted generation drift, the three credential-free test layers, docs generation; a secret-gated live smoke job (skipped with a notice otherwise; never the gated lifecycle); a weekly `spec-drift` job that fetches, compares with the pin and opens a labelled issue. The web deploy workflows build the site from `main` once enabled.
`.github/workflows/build-and-test.yml`: on push / PR - `npm ci`, `stackql/setup-stackql`, pin verification (warns on drift), the build steps, a hard failure on uncommitted generation drift, the three credential-free test layers, docs generation; a live smoke job that is skipped at the job level on pull requests, until a spec is pinned and until the secrets are configured (never the gated lifecycle); a weekly `spec-drift` job that fetches, compares with the pin and opens a labelled issue. The web deploy workflows build the site from `main` once enabled.

## Authentication reference

Expand Down
Loading
Loading