diff --git a/AGENTS.md b/AGENTS.md index 453b5cf..be5e214 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -43,7 +43,33 @@ built ahead of time into files and served by Cloudflare. - Every page has a title and a description of 160 characters or fewer, its own link preview (`src/pages/og/`), and structured data. - Each docs page is also Markdown at its address plus `.md`; `llms.txt` and - `llms-full.txt` are generated from the docs. + `llms-full.txt` are generated from the docs. The site's own pages get their + Markdown after the build, from their `
` (`src/lib/page-markdown.mjs`, + run by `src/lib/agent-files.mjs`), which also writes the docs' search index. + +## For AI agents + +`src/lib/agents.ts` holds what the site offers agents; the docs page about it +is `/docs/tools/agents`. + +- `/docs/mcp`: an MCP server over the docs (`src/worker.ts`), with + `search_docs`, `read_page` and `list_pages`. It reads the build's files: + each page's `.md`, `llms.txt`, and `docs/mcp/search.json`, which nobody + else can fetch. A new tool raises `docsMcp.version`. +- The same tools in the browser, through WebMCP (`src/scripts/webmcp.ts`), + on every page. +- `/.well-known/`: the API catalog, the MCP server card, the AI catalog and + the skills index (`src/pages/.well-known/`). The skill itself is + `src/skills/firenook/SKILL.md`: facts from the docs only, like every page. +- `/auth.md` says nothing needs an account. `robots.txt` welcomes every + crawler with Content Signals; the home page's `Link` header and every + page's `` point to the rest. +- On the zone: two URL rewrite rules give a page's Markdown to a request + with `Accept: text/markdown` (`/` to `/index.md`, any other page to its + path plus `.md`), and DNS has `_mcp._agents`, `_index._agents` (SVCB) and + `_catalog._agents` (TXT) for DNS-AID and ARD. +- `npm run check:site` checks that every page has its Markdown, that every + search result leads to a page and heading, and the skill's digest. ## Development @@ -57,11 +83,12 @@ npm run package # after a build: dist/ as Cloudflare's build output (cf-wrang npm run deploy # cf deploy --prebuilt; CI does this on every push to main ``` -The site is a Worker with no code: only the files in `dist/`, so every request -is a static asset and costs nothing. Never add an adapter or a Worker -entrypoint (`cf init` on this project adds the Astro Cloudflare adapter; don't -run it). `www.firenook.dev` redirects to `firenook.dev` by a zone redirect -rule; the Worker has no `workers.dev` address. +The site is the files in `dist/`, served as static assets, which cost +nothing. Its Worker's code (`src/worker.ts`) runs for `/docs/mcp` only +(`runWorkerFirst` in `cloudflare.config.ts`); keep every other address static. +Never add an adapter (`cf init` on this project adds the Astro Cloudflare +adapter; don't run it). `www.firenook.dev` redirects to `firenook.dev` by a +zone redirect rule; the Worker has no `workers.dev` address. A new release: copy the engine's changelog and compatibility facts, add its npm publish time to `published` in `src/lib/changelog.ts` (and to `measured` diff --git a/README.md b/README.md index 7dfac7d..c190dcb 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,8 @@ npm run check:site # then every link, anchor, title and description in dist/ A push to `main` deploys the site to firenook.dev (`.github/workflows/deploy.yml`); a pull request builds and checks it. The -site is static files on Cloudflare, with no code running per request: +site is static files on Cloudflare; the one address that runs code is +`/docs/mcp`, an MCP server over the docs (`src/worker.ts`). `cloudflare.config.ts` and `wrangler.config.ts` describe it. To deploy by hand, after `cf auth login`: diff --git a/astro.config.mjs b/astro.config.mjs index cc12293..ed15700 100644 --- a/astro.config.mjs +++ b/astro.config.mjs @@ -4,6 +4,7 @@ import starlight from '@astrojs/starlight'; import sitemap from '@astrojs/sitemap'; import starlightLinksValidator from 'starlight-links-validator'; import { sidebar } from './src/lib/docs-sidebar.mjs'; +import { agentFiles } from './src/lib/agent-files.mjs'; // firenook.dev: every page is built ahead of time, as a file, and Cloudflare // serves them. The docs, at /docs, are Starlight: Markdown in @@ -86,11 +87,17 @@ export default defineConfig({ '/security', '/trademark', '/*.txt', + '/*.md', '/docs/**/*.md', + '/docs/mcp', + '/docs/mcp/**', + '/.well-known/**', ], }), ], }), sitemap({ filter: (page) => !page.includes('/404') }), + // Each page as Markdown, and the docs' search index (src/lib/agent-files.mjs). + agentFiles(), ], }); diff --git a/cloudflare.config.ts b/cloudflare.config.ts index ea25a1e..ef74ffd 100644 --- a/cloudflare.config.ts +++ b/cloudflare.config.ts @@ -1,19 +1,24 @@ -// firenook.dev on Cloudflare: a Worker with no code of its own, only the -// files in dist/ (wrangler.config.ts). Every request is a static asset, which -// Cloudflare serves without running a Worker. /docs/install is served from -// docs/install.html, /docs/install.html redirects to /docs/install, and an -// address with no file gets 404.html with a 404 status. public/_headers sets -// the headers. www.firenook.dev is sent to firenook.dev by a redirect rule. -import { defineConfig } from 'cf/config'; +// firenook.dev on Cloudflare: the files in dist/ (wrangler.config.ts), served +// without running code, and one Worker for one address: /docs/mcp, the docs' +// MCP server (src/worker.ts). Only requests under /docs/mcp reach it +// (runWorkerFirst); every other request is a static asset. /docs/install is +// served from docs/install.html, /docs/install.html redirects to +// /docs/install, and an address with no file gets 404.html with a 404 +// status. public/_headers sets the headers. www.firenook.dev is sent to +// firenook.dev by a redirect rule. +import { bindings, defineConfig } from 'cf/config'; export default defineConfig({ worker: { name: 'firenook-dev', compatibilityDate: '2026-10-19', + entrypoint: './src/worker.ts', assets: { htmlHandling: 'auto-trailing-slash', notFoundHandling: '404-page', + runWorkerFirst: ['/docs/mcp', '/docs/mcp/*'], }, + env: { ASSETS: bindings.assets() }, domains: ['firenook.dev', 'www.firenook.dev'], // The site has one address: no copy on workers.dev. workersDev: false, diff --git a/package-lock.json b/package-lock.json index 935af53..8af65e9 100644 --- a/package-lock.json +++ b/package-lock.json @@ -24,6 +24,8 @@ "@resvg/resvg-js": "^2.6.2", "@types/node": "^24.19.2", "cf": "1.0.0-beta.21", + "hast-util-to-mdast": "^10.1.2", + "mdast-util-gfm": "^3.1.0", "satori": "^0.63.2", "typescript": "^6.0.3", "wrangler": "4.152.0" @@ -412,6 +414,7 @@ "resolved": "https://registry.npmjs.org/@astrojs/starlight/-/starlight-0.42.6.tgz", "integrity": "sha512-Cfs7GdgzgNRCauDxQVsFUVL1thrGHK/rQdbv/SURv/xhGBXx+j5ypOHW9ENPraQxbIzUC8KAnJV7L9mvEVRuyQ==", "license": "MIT", + "peer": true, "dependencies": { "@astrojs/markdown-satteri": "^0.4.0", "@astrojs/mdx": "^8.0.0", @@ -958,6 +961,7 @@ "integrity": "sha512-zLpS5asjEb7lq8jYLq37N6XKaE41DIexlY1rF/z4/tIl3wo13Sqm28fRyfIsKZD+NZ8mM5RoKkpW/rBcuoSZSg==", "devOptional": true, "license": "MIT", + "peer": true, "dependencies": { "@emnapi/wasi-threads": "1.2.3", "tslib": "^2.4.0" @@ -969,6 +973,7 @@ "integrity": "sha512-Xz4Tpyki7XyrpbUK1jR1AhdAdaXyhhY4lZ3neLodmhpuWfy2PAQN5B46sAiU4liOXGLkHypn/qU+jvfWSCYYLA==", "devOptional": true, "license": "MIT", + "peer": true, "dependencies": { "tslib": "^2.4.0" } @@ -990,6 +995,7 @@ "cpu": [ "ppc64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1006,6 +1012,7 @@ "cpu": [ "arm" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1022,6 +1029,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1038,6 +1046,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1054,6 +1063,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1070,6 +1080,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1086,6 +1097,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1102,6 +1114,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1118,6 +1131,7 @@ "cpu": [ "arm" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1134,6 +1148,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1150,6 +1165,7 @@ "cpu": [ "ia32" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1166,6 +1182,7 @@ "cpu": [ "loong64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1182,6 +1199,7 @@ "cpu": [ "mips64el" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1198,6 +1216,7 @@ "cpu": [ "ppc64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1214,6 +1233,7 @@ "cpu": [ "riscv64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1230,6 +1250,7 @@ "cpu": [ "s390x" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1246,6 +1267,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1262,6 +1284,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1278,6 +1301,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1294,6 +1318,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1310,6 +1335,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1326,6 +1352,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1342,6 +1369,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1358,6 +1386,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1374,6 +1403,7 @@ "cpu": [ "ia32" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1390,6 +1420,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -2998,6 +3029,7 @@ "integrity": "sha512-Thbli+OlOj+iMPYFBVBfJ3OmCAnaSyNn4M1vz9T6Gka5Jt9ba/HIR56joy65tY6kx/FCF5VXNB819Y7/GUrBGA==", "dev": true, "license": "MIT", + "peer": true, "dependencies": { "fast-deep-equal": "^3.1.3", "fast-uri": "^3.0.1", @@ -3150,6 +3182,7 @@ "resolved": "https://registry.npmjs.org/astro/-/astro-7.4.1.tgz", "integrity": "sha512-mzWNEWSef1VQDgFku6Gp0v8Qx53NOkNlOIDQcX9ovxAZ1QXic6cks/FFo+gxPM9anXuat4X1FhB61KTk3XFvgg==", "license": "MIT", + "peer": true, "dependencies": { "@astrojs/compiler-rs": "^0.5.1", "@astrojs/internal-helpers": "0.13.0", @@ -3921,6 +3954,7 @@ "devOptional": true, "hasInstallScript": true, "license": "MIT", + "peer": true, "bin": { "esbuild": "bin/esbuild" }, @@ -3973,6 +4007,19 @@ "dev": true, "license": "MIT" }, + "node_modules/escape-string-regexp": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/escape-string-regexp/-/escape-string-regexp-5.0.0.tgz", + "integrity": "sha512-/veY75JbMK4j1yjvuUxuVsiS/hr/4iHs9FTT6cgTexxdE0Ly/glccBAkloH/DofkjRbZU3bnoj38mOmhkZ0lHw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/eventemitter3": { "version": "5.0.4", "resolved": "https://registry.npmjs.org/eventemitter3/-/eventemitter3-5.0.4.tgz", @@ -4445,6 +4492,33 @@ "url": "https://opencollective.com/unified" } }, + "node_modules/hast-util-to-mdast": { + "version": "10.1.2", + "resolved": "https://registry.npmjs.org/hast-util-to-mdast/-/hast-util-to-mdast-10.1.2.tgz", + "integrity": "sha512-FiCRI7NmOvM4y+f5w32jPRzcxDIz+PUqDwEqn1A+1q2cdp3B8Gx7aVrXORdOKjMNDQsD1ogOr896+0jJHW1EFQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "@types/mdast": "^4.0.0", + "@ungap/structured-clone": "^1.0.0", + "hast-util-phrasing": "^3.0.0", + "hast-util-to-html": "^9.0.0", + "hast-util-to-text": "^4.0.0", + "hast-util-whitespace": "^3.0.0", + "mdast-util-phrasing": "^4.0.0", + "mdast-util-to-hast": "^13.0.0", + "mdast-util-to-string": "^4.0.0", + "rehype-minify-whitespace": "^6.0.0", + "trim-trailing-lines": "^2.0.0", + "unist-util-position": "^5.0.0", + "unist-util-visit": "^5.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, "node_modules/hast-util-to-string": { "version": "3.0.1", "resolved": "https://registry.npmjs.org/hast-util-to-string/-/hast-util-to-string-3.0.1.tgz", @@ -5034,6 +5108,17 @@ "source-map-js": "^1.2.1" } }, + "node_modules/markdown-table": { + "version": "3.0.4", + "resolved": "https://registry.npmjs.org/markdown-table/-/markdown-table-3.0.4.tgz", + "integrity": "sha512-wiYz4+JrLyb/DqW2hkFJxP7Vd7JuTDm77fvbM8VfEQdmSMqcImWeeRbHwZjBjIFki/VaMK2BhFi7oUUZeM5bqw==", + "dev": true, + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, "node_modules/mdast-util-directive": { "version": "3.1.1", "resolved": "https://registry.npmjs.org/mdast-util-directive/-/mdast-util-directive-3.1.1.tgz", @@ -5056,6 +5141,23 @@ "url": "https://opencollective.com/unified" } }, + "node_modules/mdast-util-find-and-replace": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/mdast-util-find-and-replace/-/mdast-util-find-and-replace-3.0.3.tgz", + "integrity": "sha512-xPgpDNl0/OXHsI7TlaIs22lWnH3KlpvZdXG1ET/m/YT2Hhdkx8lJV4kLphE6l9simyM4KPDFBwas08BgEob6Jw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "escape-string-regexp": "^5.0.0", + "unist-util-is": "^6.0.0", + "unist-util-visit-parents": "^6.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, "node_modules/mdast-util-from-markdown": { "version": "2.1.0", "resolved": "https://registry.npmjs.org/mdast-util-from-markdown/-/mdast-util-from-markdown-2.1.0.tgz", @@ -5080,6 +5182,113 @@ "url": "https://opencollective.com/unified" } }, + "node_modules/mdast-util-gfm": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/mdast-util-gfm/-/mdast-util-gfm-3.1.0.tgz", + "integrity": "sha512-0ulfdQOM3ysHhCJ1p06l0b0VKlhU0wuQs3thxZQagjcjPrlFRqY215uZGHHJan9GEAXd9MbfPjFJz+qMkVR6zQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "mdast-util-from-markdown": "^2.0.0", + "mdast-util-gfm-autolink-literal": "^2.0.0", + "mdast-util-gfm-footnote": "^2.0.0", + "mdast-util-gfm-strikethrough": "^2.0.0", + "mdast-util-gfm-table": "^2.0.0", + "mdast-util-gfm-task-list-item": "^2.0.0", + "mdast-util-to-markdown": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-gfm-autolink-literal": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/mdast-util-gfm-autolink-literal/-/mdast-util-gfm-autolink-literal-2.0.1.tgz", + "integrity": "sha512-5HVP2MKaP6L+G6YaxPNjuL0BPrq9orG3TsrZ9YXbA3vDw/ACI4MEsnoDpn6ZNm7GnZgtAcONJyPhOP8tNJQavQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "ccount": "^2.0.0", + "devlop": "^1.0.0", + "mdast-util-find-and-replace": "^3.0.0", + "micromark-util-character": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-gfm-footnote": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/mdast-util-gfm-footnote/-/mdast-util-gfm-footnote-2.1.0.tgz", + "integrity": "sha512-sqpDWlsHn7Ac9GNZQMeUzPQSMzR6Wv0WKRNvQRg0KqHh02fpTz69Qc1QSseNX29bhz1ROIyNyxExfawVKTm1GQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "devlop": "^1.1.0", + "mdast-util-from-markdown": "^2.0.0", + "mdast-util-to-markdown": "^2.0.0", + "micromark-util-normalize-identifier": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-gfm-strikethrough": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/mdast-util-gfm-strikethrough/-/mdast-util-gfm-strikethrough-2.0.1.tgz", + "integrity": "sha512-OuJHqvr455pwu2OaOrir7dbzqs4jlWKOtlu9L6GjcIjjQwNyw02VntQsr/Ek6/A13vgnXfUhBekWRUP1OkNivw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "mdast-util-from-markdown": "^2.0.0", + "mdast-util-to-markdown": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-gfm-table": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/mdast-util-gfm-table/-/mdast-util-gfm-table-2.0.0.tgz", + "integrity": "sha512-78UEvebzz/rJIxLvE7ZtDd/vIQ0RHv+3Mh5DR96p7cS7HsBhYIICDBCu8csTNWNO6tBWfqXPWekRuj2FNOGOZg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "devlop": "^1.0.0", + "markdown-table": "^3.0.0", + "mdast-util-from-markdown": "^2.0.0", + "mdast-util-to-markdown": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, + "node_modules/mdast-util-gfm-task-list-item": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/mdast-util-gfm-task-list-item/-/mdast-util-gfm-task-list-item-2.0.0.tgz", + "integrity": "sha512-IrtvNvjxC1o06taBAVJznEnkiHxLFTzgonUdy8hzFVeDun0uTjxxrRGVaNFqkU1wJR3RBPEfsxmU6jDWPofrTQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/mdast": "^4.0.0", + "devlop": "^1.0.0", + "mdast-util-from-markdown": "^2.0.0", + "mdast-util-to-markdown": "^2.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, "node_modules/mdast-util-mdx-jsx": { "version": "3.2.0", "resolved": "https://registry.npmjs.org/mdast-util-mdx-jsx/-/mdast-util-mdx-jsx-3.2.0.tgz", @@ -6040,6 +6249,7 @@ } ], "license": "MIT", + "peer": true, "dependencies": { "nanoid": "^3.3.20", "picocolors": "^1.1.1", @@ -6100,6 +6310,7 @@ "integrity": "sha512-skGHTH1Yb9QvaTC4hWnRTzEYtCpApF0UMBPz+t/0ZPzP8ihefo/btKd+QA7pUE0fEwA1SbS/NmhyEcstdXOIdA==", "dev": true, "license": "MIT", + "peer": true, "bin": { "prettier": "bin/prettier.cjs" }, @@ -6190,6 +6401,21 @@ "expressive-code": "^0.44.2" } }, + "node_modules/rehype-minify-whitespace": { + "version": "6.0.2", + "resolved": "https://registry.npmjs.org/rehype-minify-whitespace/-/rehype-minify-whitespace-6.0.2.tgz", + "integrity": "sha512-Zk0pyQ06A3Lyxhe9vGtOtzz3Z0+qZ5+7icZ/PL/2x1SHPbKao5oB/g/rlc6BCTajqBb33JcOe71Ye1oFsuYbnw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/hast": "^3.0.0", + "hast-util-minify-whitespace": "^1.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/unified" + } + }, "node_modules/remark-directive": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/remark-directive/-/remark-directive-4.0.0.tgz", @@ -6695,6 +6921,17 @@ "url": "https://github.com/sponsors/wooorm" } }, + "node_modules/trim-trailing-lines": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/trim-trailing-lines/-/trim-trailing-lines-2.1.0.tgz", + "integrity": "sha512-5UR5Biq4VlVOtzqkm2AZlgvSlDJtME46uV0br0gENbwN4l5+mMKT4b9gJKqWtuL2zAIqajGJGuvbCbcAJUZqBg==", + "dev": true, + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, "node_modules/trough": { "version": "2.2.0", "resolved": "https://registry.npmjs.org/trough/-/trough-2.2.0.tgz", @@ -6725,6 +6962,7 @@ "integrity": "sha512-y2TvuxSZPDyQakkFRPZHKFm+KKVqIisdg9/CZwm9ftvKXLP8NRWj38/ODjNbr43SsoXqNuAisEf1GdCxqWcdBw==", "devOptional": true, "license": "Apache-2.0", + "peer": true, "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" @@ -6782,6 +7020,7 @@ "integrity": "sha512-i7qRCmY42zmCwnYlh9H2SvLEypEFGye5iRmEMKjcGi7zk9UquigRjFtTLz0TYqr0ZGLZhaMHl/foy1bZR+Cwlw==", "dev": true, "license": "MIT", + "peer": true, "dependencies": { "pathe": "^2.0.3" } @@ -7083,6 +7322,7 @@ "resolved": "https://registry.npmjs.org/vite/-/vite-8.3.4.tgz", "integrity": "sha512-IftLE8phJhdNRIGNEV4JCCvfNScoZQVfcXaN8TQropWGJtOHcu+iAB9pHcuGvmEb/kddw0LBe/jS8ztdC6xdiQ==", "license": "MIT", + "peer": true, "dependencies": { "lightningcss": "^1.33.0", "picomatch": "^4.0.7", @@ -7618,6 +7858,7 @@ "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.9.1.tgz", "integrity": "sha512-3NxN8+78OdzbT7C/WjGsyfPAtJaN3FNDsWxv7Y7mcDsT/oOmgW8BpyQQFFBnvZE3j9Y2Sdz1ULFLezL7Eb2yFw==", "license": "ISC", + "peer": true, "bin": { "yaml": "bin.mjs" }, diff --git a/package.json b/package.json index 840442f..7ee2030 100644 --- a/package.json +++ b/package.json @@ -28,6 +28,8 @@ "@resvg/resvg-js": "^2.6.2", "@types/node": "^24.19.2", "cf": "1.0.0-beta.21", + "hast-util-to-mdast": "^10.1.2", + "mdast-util-gfm": "^3.1.0", "satori": "^0.63.2", "typescript": "^6.0.3", "wrangler": "4.152.0" diff --git a/public/_headers b/public/_headers index 4ea05ef..86ccc29 100644 --- a/public/_headers +++ b/public/_headers @@ -1,4 +1,5 @@ /* + Vary: Accept X-Content-Type-Options: nosniff Referrer-Policy: strict-origin-when-cross-origin Permissions-Policy: camera=(), microphone=(), geolocation=(), browsing-topics=() @@ -11,7 +12,20 @@ /og/* Cache-Control: public, max-age=86400 -# The Markdown copies of the docs are for AI tools; search results show the pages. +# Where an agent finds what the site offers it (RFC 8288, RFC 9727). +/ + Link: ; rel="api-catalog", ; rel="service-doc", ; rel="describedby"; type="text/plain", ; rel="alternate"; type="text/markdown" + +/.well-known/* + Access-Control-Allow-Origin: * + X-Robots-Tag: noindex + +/.well-known/api-catalog + Content-Type: application/linkset+json + +# The pages as Markdown are for AI tools; search results show the pages. A +# request for a page with Accept: text/markdown gets them too, by a zone +# rewrite rule (AGENTS.md); Vary: Accept above keeps caches apart. /*.md Content-Type: text/markdown; charset=utf-8 X-Robots-Tag: noindex diff --git a/public/auth.md b/public/auth.md new file mode 100644 index 0000000..9877f1d --- /dev/null +++ b/public/auth.md @@ -0,0 +1,34 @@ +# firenook.dev auth.md + +firenook.dev needs no account, no registration and no credentials. Every +page, the docs as Markdown, llms.txt and the docs' MCP server are public, and +an agent uses them as they are, on behalf of anyone. + +## Registration + +None. There is nothing to sign up for, and no endpoint creates an account. + +## Signing in + +None. No address on firenook.dev asks for a token, an API key or a cookie, +and none sets one. + +## What an agent can use + +- The docs' MCP server, `https://firenook.dev/docs/mcp`, over Streamable + HTTP, with no `Authorization` header. Its card is at + `https://firenook.dev/docs/mcp/server-card`. +- Every page as Markdown: its address plus `.md`, or the page's address with + `Accept: text/markdown`. +- The list of pages, `https://firenook.dev/llms.txt`, and the Firenook skill, + `https://firenook.dev/.well-known/agent-skills/firenook/SKILL.md`. + +## Firenook itself + +Firenook runs the Firebase emulators on your own machine, and it has no +account either. Its Authentication emulator's test users and tokens are +described at `https://firenook.dev/docs/services/auth`. + +## Security + +How to report a vulnerability: `https://firenook.dev/security`. diff --git a/public/robots.txt b/public/robots.txt index 2e9a359..26e2a4f 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -1,5 +1,23 @@ -# Search engines and AI tools are welcome to everything here. +# Search engines and AI tools are welcome to everything here: to index it, to +# answer with it, and to learn from it (Content Signals, contentsignals.org). User-agent: * +Content-Signal: search=yes, ai-input=yes, ai-train=yes +Allow: / + +# The AI crawlers and agents by name, with the same welcome. +User-agent: GPTBot +User-agent: OAI-SearchBot +User-agent: ChatGPT-User +User-agent: ClaudeBot +User-agent: Claude-SearchBot +User-agent: Claude-User +User-agent: Google-Extended +User-agent: Applebot-Extended +User-agent: PerplexityBot +User-agent: Perplexity-User +User-agent: meta-externalagent +User-agent: CCBot +Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Sitemap: https://firenook.dev/sitemap-index.xml diff --git a/scripts/check-site.mjs b/scripts/check-site.mjs index 56c1a09..1dd900e 100644 --- a/scripts/check-site.mjs +++ b/scripts/check-site.mjs @@ -1,9 +1,12 @@ // Checks the built site in dist/: every internal link and anchor leads // somewhere, every page has one title and a description of 160 characters -// or fewer, titles are unique, and every page's link preview exists. +// or fewer, titles are unique, and every page's link preview exists. For +// agents: every page has its Markdown, every result the docs' MCP server can +// give leads to a page and heading, and the skill matches its digest. // // npm run build && node scripts/check-site.mjs +import { createHash } from 'node:crypto'; import { readFileSync, readdirSync, existsSync, statSync } from 'node:fs'; import { join, relative } from 'node:path'; @@ -69,6 +72,21 @@ for (const file of pages) { for (const [title, where] of titles) if (where.length > 1) problems.push(`title "${title}" on ${where.join(', ')}`); +for (const file of pages) { + if (!file.endsWith('/404.html') && !existsSync(file.replace(/\.html$/, '.md'))) problems.push(`/${relative(dist, file)}: no Markdown`); +} +const index = JSON.parse(readFileSync(join(dist, 'docs/mcp/search.json'), 'utf8')); +for (const section of index.flatMap((page) => page.sections)) { + const [path, anchor] = section.url.slice(site.length).split('#'); + const file = target(path || '/'); + if (!file) problems.push(`docs search: no page for ${section.url}`); + else if (anchor && !idsOf(file).has(anchor)) problems.push(`docs search: missing anchor ${section.url}`); +} +for (const skill of JSON.parse(readFileSync(join(dist, '.well-known/agent-skills/index.json'), 'utf8')).skills) { + const digest = `sha256:${createHash('sha256').update(readFileSync(join(dist, skill.url))).digest('hex')}`; + if (digest !== skill.digest) problems.push(`skill ${skill.name}: digest ${skill.digest}, served file ${digest}`); +} + console.log(`${pages.length} pages, ${links} internal links checked`); if (problems.length) { console.log([...new Set(problems)].join('\n')); diff --git a/src/components/Seo.astro b/src/components/Seo.astro index 6d1da33..f3dfd7d 100644 --- a/src/components/Seo.astro +++ b/src/components/Seo.astro @@ -24,6 +24,7 @@ const og = new URL(image ?? `/og/${slug}.png`, site.url).href; {title} {noindex ? : } +{!noindex && } diff --git a/src/components/docs/Head.astro b/src/components/docs/Head.astro index 06a610b..7379820 100644 --- a/src/components/docs/Head.astro +++ b/src/components/docs/Head.astro @@ -22,3 +22,6 @@ const fonts = [ + diff --git a/src/content/docs/docs/tools/agents.md b/src/content/docs/docs/tools/agents.md new file mode 100644 index 0000000..5b3f03f --- /dev/null +++ b/src/content/docs/docs/tools/agents.md @@ -0,0 +1,91 @@ +--- +title: Use these docs from an agent +description: Give a coding agent the Firenook docs through an MCP server that searches and reads them, every page as Markdown, llms.txt, and a skill. +--- + +Everything on firenook.dev is public and needs no sign-in, so an agent reads +it as you do. There are four ways to hand it the docs. + +## The docs' MCP server + +`https://firenook.dev/docs/mcp` is an MCP server over these docs, on +Streamable HTTP, with three tools: + +| Tool | What it does | +| --- | --- | +| `search_docs` | Finds the sections that answer a question, with their addresses | +| `read_page` | Returns a page as Markdown | +| `list_pages` | Lists every page, with a line on what each covers | + +It is not `firenook mcp`. That server works on the emulators running on your +machine ([connect your coding agent](/docs/tools/mcp)); this one reads the +docs. + +Add it to Claude Code: + +```sh +claude mcp add --transport http firenook-docs https://firenook.dev/docs/mcp +``` + +To Cursor, in `.cursor/mcp.json`: + +```json title=".cursor/mcp.json" +{ + "mcpServers": { + "firenook-docs": { "url": "https://firenook.dev/docs/mcp" } + } +} +``` + +To VS Code, in `.vscode/mcp.json`: + +```json title=".vscode/mcp.json" +{ + "servers": { + "firenook-docs": { "type": "http", "url": "https://firenook.dev/docs/mcp" } + } +} +``` + +## Pages as Markdown + +Every page is also Markdown, at its address plus `.md`: + +```sh +curl https://firenook.dev/docs/get-started/install.md +``` + +Or ask for the page itself as Markdown: + +```sh +curl -H 'Accept: text/markdown' https://firenook.dev/docs/get-started/install +``` + +[llms.txt](https://firenook.dev/llms.txt) lists every page with a line on +what it covers, and [llms-full.txt](https://firenook.dev/llms-full.txt) holds +all of the docs in one file. + +## The Firenook skill + +The skill teaches an agent to install Firenook, switch a project from the +Firebase emulators, keep data between runs and run tests. It is one file, +`https://firenook.dev/.well-known/agent-skills/firenook/SKILL.md`. For +Claude Code, save it in the project: + +```sh +mkdir -p .claude/skills/firenook +curl -o .claude/skills/firenook/SKILL.md https://firenook.dev/.well-known/agent-skills/firenook/SKILL.md +``` + +Agents that look for skills on a site find it listed at +`https://firenook.dev/.well-known/agent-skills/index.json`. + +## In the browser + +An agent that works in the browser through WebMCP finds the same three tools +on every page, and `open_page` to go to another. + +## Crawlers + +`robots.txt` lets every crawler read every page, and its Content Signals say +the pages may be used for search, for answers and for training. diff --git a/src/layouts/Base.astro b/src/layouts/Base.astro index c8cf854..8ddb794 100644 --- a/src/layouts/Base.astro +++ b/src/layouts/Base.astro @@ -57,6 +57,9 @@ const fonts = [