Skip to content

feat(web): llms.txt, llms-full.txt, llms-small.txt and per-page Markdown for guides and API - #4356

Merged
bpamiri merged 1 commit into
developfrom
peter/llms-txt
Oct 5, 2026
Merged

bpamiri merged 1 commit into
developfrom
peter/llms-txt

Conversation

@bpamiri

@bpamiri bpamiri commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds plain-text docs for language models (llms.txt) to the guides and API sites, plus a root index on wheels.dev. Today guides.wheels.dev/llms.txt and api.wheels.dev/llms.txt are 404. wheels.dev/llms.txt answers 200 only because the landing host serves the home page for unknown paths; that's fixed in a separate PR.

Served on both guides.wheels.dev and api.wheels.dev:

URL content
/llms.txt Index: what Wheels is, a few CFML-not-Ruby notes, the documentation sets, every current-version page with a link to its Markdown, plus pointers to the other docs site, the blog and packages.
/llms-full.txt Every current-version page as Markdown, in one file.
/llms-small.txt An outline of the current version: each page's title, Markdown link, description and section headings. Sized for an assistant's context; it fetches the page Markdown it needs.
/llms-txt/<version>.txt Every page of one docs version (archived, current and in development), in one file.
<page URL>.md One page as Markdown, for example /v4-1-0/basics/routing.md.

Plus wheels.dev/llms.txt, a static index (web/sites/landing/public/llms.txt). It links the guides and API llms.txt, and lists the blog and packages as sections.

How it's built

  • No new dependencies. starlight-llms-txt@0.10.0, the only line compatible with the Astro 6.4.6 these sites actually build with, ignores exclude for llms-full.txt, which then held all five guides versions (7.6 MB measured). Newer releases need Starlight 0.41 or later.
  • Plain Astro endpoints instead: src/pages/llms.txt.ts, llms-full.txt.ts, llms-small.txt.ts, llms-txt/[slug].txt.ts and [...slug].md.ts in each site. They're thin wrappers over the shared @wheels-dev/ui/data/llms (packages/ui/src/data/llms.ts). (_llms-txt/ couldn't be used: Astro ignores page directories that start with _.)
  • Current version: taken from the current entry in versions.ts, so when a version flip makes 4.2 current (draft docs(web): make the 4.2 guides, API reference and home page links current #4322), all of this follows with no further edit.
  • Excluded from the index and bundles: draft pages and unlisted ones (sidebar.hidden: true or pagefind: false). The site root index page is a version list, so it's left out too.
  • Page Markdown: title and description, then the page source with MDX import lines removed. Components such as <Aside> stay as written.

CI check: web/scripts/check-llms.mjs. It runs in web-deploy.yml after the guides and API builds, before deploying. It fails when:

  • a file is missing or empty;
  • a link in llms.txt to the site doesn't resolve to a built file;
  • a non-current version appears in llms-full.txt or llms-small.txt;
  • a built docs page has no .md, or a .md has no page. Meta-refresh redirect stubs are skipped.

Its fixture tests, web/scripts/check-llms.test.mjs (7 cases), run in web-scripts-tests.yml. They cover a consistent build passing, a version leak into each of full and small, a broken link, a missing and an orphan .md, missing and empty files, and the versions.ts parse.

Measured (develop + this branch; current version v4-1-0)

llms.txt llms-full.txt llms-small.txt page .md version sets
guides 43 KB 1.54 MB 76 KB 851 5 (632 KB to 1.66 MB each)
api 66 KB 0.91 MB 84 KB 3135 10 (372 to 905 KB each)

llms-small.txt is about 20k tokens per site. wheels.dev/llms.txt is 1.3 KB.

node scripts/check-llms.mjs sites/guides/dist guides and … sites/api/dist api pass on the real builds. The fixture tests are 7/7. pnpm build passes for guides, API and landing, and Prettier is clean. actionlint shows no new findings (the 2 existing shellcheck notes are in detect-scope).

Noticed while checking, not changed here: the v4-1-0 guides landing page (v4-1-0/index.mdx) is titled "Wheels 4.0".

🤖 Generated with Claude Code

…own for guides and API

Serve plain-text docs for language models from the guides and API sites:
an llms.txt index, llms-full.txt (every current-version page),
llms-small.txt (an outline of the current version), one set per docs
version under /llms-txt/, and every page as Markdown at <page URL>.md.
The current version comes from versions.ts. Built by plain Astro
endpoints over the content collection, with no new dependencies. Add a
wheels.dev/llms.txt index, and a CI check that fails the deploy when a
file is missing, a link doesn't resolve, a page lacks its .md, or a
non-current version leaks into llms-full / llms-small.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bpamiri

bpamiri commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

rev1-r3 review @ 87890d7: code APPROVE (final sign-off after PR CI, including the guides and API builds with the new Check llms files step)

  • Current-version scoping. currentVersion() reads the single status: 'current' entry from versions.ts, and throws if there isn't one, so a missing current fails the build instead of silently emitting nothing. llms.txt, llms-full.txt and llms-small.txt all go through docsOfVersion(docs, current.slug), which drops drafts and unlisted pages (pagefind: false or sidebar.hidden) and keeps the version's landing page first. So when the version flip makes 4.2 current, all three follow automatically. The per-version llms-txt/<slug>.txt sets cover every versions.ts entry, archived and in-development included, which is the intended split.
  • Page Markdown. [...slug].md.ts emits one .md per non-draft entry. markdownTwin() writes the title and description followed by the source, with whole-line MDX import lines removed (anchored at ^import … from '…';?$, so prose that merely starts with "import" isn't touched). Components stay as written, and llms.txt tells consumers how to fetch .md for any page.
  • llms-small.txt's outline skips headings inside fenced code (it tracks the opening fence marker, so a longer closing fence still closes it), so ## lines in code samples don't become fake sections.
  • The CI check is placed and scoped right. Check llms files runs only for the guides/api matrix entries, after the build and before the Cloudflare deploy, so a broken or leaking llms-full/llms-small never ships. Its 7 fixture tests are picked up by web-scripts-tests.yml's existing web/scripts/**/*.test.mjs glob.
  • The engine list ("Lucee, Adobe ColdFusion, BoxLang and RustCFML") matches what the landing page already states.

Not blocking:

  1. Every URL in these files uses the public origin (https://guides.wheels.dev/…), including in the offline docs bundle the framework serves at /wheels-docs/guides/. So the bundle's llms.txt points readers back online. That's fine for the online use case, but worth knowing if the bundle is ever expected to work fully offline for an assistant.
  2. The .md twins and version sets add about 4,000 small files and roughly 15 MB to the two Pages deploys (the per-file and file-count limits are comfortably met). The deploy time increase will show in the first run.

@bpamiri

bpamiri commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

Review @ 87890d7 (text: llms.ts SITE_TEXT / DETAILS and the landing llms.txt): APPROVE on wording

rev1-r3 has the build side, so this covers the text only.

  • The facts are right:

    • the engine list (Lucee, Adobe ColdFusion, BoxLang, RustCFML) matches the v4-1-0 cfml-engines guide;
    • "models extend Model, controllers extend Controller, associations / validations / callbacks in config()";
    • "all positional or all named arguments, never a mix" is the framework's own rule (the New master #1 anti-pattern in the repo guide);
    • "model finders return query objects by default, not arrays".

    These are the three mistakes assistants most often make with Wheels code, so they're the right lines to put in front of a model.

  • The links resolve. blog.wheels.dev, packages.wheels.dev and blog.wheels.dev/rss.xml (served from the blog's rss.xml.js) all return 200. The command wheels packages add <name> is the 4.x spelling.

  • The structure follows llmstxt.org: an H1, a one-paragraph blockquote summary, detail lines, H2 link lists, and an ## Optional section for what a model can skip.

  • The index is version-neutral. It derives "Wheels " and the per-version set links from versions.ts, so a version flip needs no text edit. The landing file says "current release" rather than a number.

INFO (optional):

  • The API "what" line says every function has "examples". Some reference entries have none, so "and, where present, examples" would be exact.
  • In llmsIndex() the four DETAILS lines and the ".md" line are emitted on consecutive lines with no blank line between them. That's fine for a plain-text consumer, but rendered as Markdown they merge into one paragraph. Joining them into one paragraph, as the landing file does, or blank-separating them would make the intent explicit.

No findings.

@github-actions github-actions Bot added dependencies Pull requests that update a dependency file javascript Pull requests that update javascript code labels Oct 5, 2026
@bpamiri

bpamiri commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

rev1-r3 review @ 87890d7: APPROVE (final; PR CI checked)

PR CI on this head: 20 pass, 3 skipped, 0 failing or pending. Both docs deploys ran the new step and it passed:

  • Deploy guides: "llms check passed for guides: current version v4-1-0"
  • Deploy api: "llms check passed for api: current version v4-1-0"

So on the real builds every file exists, every llms.txt link resolves, every page has its .md, and no other version leaks into llms-full.txt / llms-small.txt.

@bpamiri
bpamiri merged commit 033f834 into develop Oct 5, 2026
23 checks passed
@bpamiri
bpamiri deleted the peter/llms-txt branch October 5, 2026 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file javascript Pull requests that update javascript code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant