diff --git a/CLAUDE.md b/CLAUDE.md index d47632b..0e6b11e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -44,6 +44,16 @@ verified go-task or platform fact that cost a session to find. (`github.job_workflow_sha`) for the toolbox action and the lock, so a workflow pin is the one pin. The include and the image float at `v2` by design: moving that tag is the rollout. +- GNU `xargs` runs its command once on empty input. A guard on the file feeding a pipe is + not a guard on what reaches `xargs`: `pages`'s `SLASH` awk drops the root, which has no + slashless key, so a run whose only dirty directory is the root sends it nothing. Every + `xargs` whose input can be filtered down to nothing takes `-r`. +- A `>-` folded block keeps the newline when a continuation line is indented further than + the lines around it. `pages`'s awk programs depend on that; a shell line meant to + continue ends with a backslash. `task check` shows what actually renders. +- A macOS disk merges directories that differ only in case. Pages drawn on a laptop from a + real listing come out short wherever upstream has two such directories (CTAN has + `obsolete/support/TeXshell/` and `texshell/`). The runner is ext4 and draws both. ## Verifying a change @@ -54,8 +64,9 @@ cd examples/proton && task image-build && task run -- task tools && task check & A verb change updates the `render.txt` files via `task render-update`; `offline` is each engine's own check. The rsync one runs over `examples/rsync/fixtures/`: the list -diff over `run-root` and `run-empty`, `retry`'s exit codes, and `prepare` and `verify` -over `tree/`, a signed subtree whose tlpdb is signed by a throwaway key pinned in the +diff over `run-root` and `run-empty`, `retry`'s exit codes, `pages` over `run-pages` +(whose `want/` is ctan's page set, matched byte for byte), `smoke`'s page read-back over +`run-smoke`, and `prepare` and `verify` over `tree/`, a signed subtree whose tlpdb is signed by a throwaway key pinned in the example. Regenerate the tree with a new key only to change its shape; the private half was never kept. The proton one runs `confirm` over `examples/proton/fixtures/`, an accepting and a refusing upload summary, and the `age` verb round trip with a throwaway diff --git a/README.md b/README.md index 5b47b9f..77ee54c 100644 --- a/README.md +++ b/README.md @@ -270,8 +270,8 @@ flowchart LR class prepare,verify,index,sm,re,rm hook ``` -Dashed verbs are hooks. `prepare` and `verify` do nothing until `TL_KEY` is set; `index` -and `smoke-mirror` do nothing until a mirror fills them. +Dashed verbs are hooks. `prepare` and `verify` do nothing until `TL_KEY` is set, `index` +until `INDEX` is set, and `smoke-mirror` until a mirror fills it. | Verb | Does | |---|---| @@ -289,8 +289,9 @@ and `smoke-mirror` do nothing until a mirror fills them. | `checkpoint` | `merge` what landed into the state and push it as one PutObject; empty staging | | `delete` | Remove the keys upstream dropped, 1,000 per call, once every batch has landed, and drop them from the state | | `reconcile` | When `due` left `.run/reconcile`: rebuild the state, delete what neither upstream, `OWN` nor the state's own directories own, then `reconciled` | -| `index` | Hook. Nothing here; a mirror that draws directory pages or a landing page replaces it | -| `smoke` | A sample of the run's keys read back through `HOST`, sizes against the listing; the tlpdb sha512 when `TL` is set; then `smoke-mirror` | +| `index` | Hook. With `INDEX` set: `pages`, then both key sets uploaded, the keys of emptied directories removed, and `.state/indexed.txt.xz` moved forward. A mirror with a landing page of its own replaces it | +| `pages` | A page for every directory the run touched, drawn from the state into staging, with a tree per depth for the slashless keys | +| `smoke` | A sample of the run's keys read back through `HOST`, sizes against the listing; the tlpdb sha512 when `TL` is set; with `INDEX`, one redrawn page under both keys; with `CANARY`, that file as `libwww-perl` against the bucket's copy; no `Content-Encoding` on the first non-empty `.tar.gz`; then `smoke-mirror` | | `smoke-mirror` | Hook. Nothing here; a mirror with more to read back defines it | | `report-engine` | Hook. The engine's rows of the run summary | | `retry` | Run a command, retrying rsync's transport exit codes with backoff; 23 (an unreadable path, skipped) and 24 (a file vanished mid-transfer) are successes | @@ -362,6 +363,24 @@ A batch that fails any check stays local; the previous good copy stays live. `sm reads `texlive.tlpdb.sha512` back through the domain afterwards and compares it with the verified copy. `tlmgr` repeats the signature check on the client. +#### Directory pages + +R2 serves no listings. With `INDEX` set, `index` draws a page for every directory a run +touched, from the state rather than from upstream, and writes it under two keys: +`/`, which a zone Transform Rule serves for `/dir/`, and the bare ``, +which serves `/dir` where a filesystem mirror would answer 301. A `` lets one +document serve both. `.state/indexed.txt.xz` records the state the pages last showed, so a +run that dies before moving it redraws the same pages next time. Deleting it redraws every +page once, which is also how a change to the markup or to `PAGE_FOOT` reaches directories +that have not changed. `reconcile` spares both keys. + +The zone rule, one per host: + + when: ends_with(http.request.uri.path, "/") + rewrite path: concat(http.request.uri.path, http.host, ".directory.index.html") + +It serves the root page for `/` too. ctan excludes `/`, which is CTAN's own `index.html`. + #### The vars A mirror sets `SOURCE`, `BUCKET` and `HOST` in its root vars, always. Everything else @@ -378,7 +397,9 @@ has an inline default, and a mirror sets only what differs: | `TL`, `TL_KEY` | empty | A signed TeX Live subtree and the fingerprint that signs it; empty, no signature checks | | `FILTER` | empty | rsync filter arguments that narrow the listing, for a mirror of a subtree | | `OWN` | empty | Bucket-root keys the mirror owns, space separated; `reconcile` never deletes them | -| `INDEX` | empty | The key suffix of the directory pages a mirror's `index` draws; set, `reconcile` spares those pages and every bare directory of the state | +| `INDEX` | empty | The key suffix of the directory pages; set, `index` draws them and `reconcile` spares them and every bare directory of the state | +| `PAGE_FOOT` | empty | The HTML every directory page closes on; `%s` is the directory's encoded path, `%%` a literal percent | +| `CANARY` | empty | A path carrying plain `http://` links or `mailto:` addresses, no character of which a URL must encode; set, `smoke` fails if the domain serves it differently from the bucket or refuses a Perl client | A var the mirror puts in its root `vars:` is fixed for every run: inside an included verb a root value shadows a `KEY=value` from the command line. One the mirror leaves to @@ -557,13 +578,15 @@ Every mirror has one S3-compatible bucket. What it holds depends on the engine. |---|---|---| | rsync | every upstream path, at the root | The mirror. A public domain serves the bucket | | rsync | `.state/applied.txt.xz` | The state: what the bucket holds, at upstream's size and mtime | -| rsync, a mirror's own | `.state/indexed.txt.xz`, `index.html` | ctan's record of what its directory pages show; tlnet's landing page, spared by `OWN` | +| rsync, with `INDEX` | `.state/indexed.txt.xz`, `/`, `` | What the directory pages last showed, and the pages under both keys | +| rsync, a mirror's own | `index.html` | tlnet's landing page, spared by `OWN` | | proton | `.state/session.tar.age` | The CLI session, encrypted. The only key | | a pipeline of its own | `.state/state.sqlite.xz.age`, `.state/history/-