Replace the GitHub Pages site with a daily cross-environment test dashboard

[gaurav]

Important

Superseded by #120, where development continues. That PR redesigns this dashboard from scratch — a nav bar and page shell, three pages instead of one, a sticky filter bar, and a promotion-drift panel — and it carries every commit below.

This PR is kept open, as a draft, only so that anything worth saving can be checked against it once #120 has merged. The pre-merge push: trigger for the dashboard workflow has moved to #120's branch, so pushes here no longer run or deploy anything. Do not add work to this branch.

The Prefix Comparator has moved into Babel (NCATSTranslator/Babel#889) and Autocomplete is moving into Babel Explorer (TranslatorSRI/babel-explorer#7), so the old site's tools are retired. In their place, https://translatorsri.github.io/babel-validation/ becomes a dashboard showing the full test suite run daily against every environment. Because test expectations are pinned to the environment where a new Babel version lands first, environments are not expected to all be green — the dashboard's purpose is to show which issues are visible in which environment.

What shipped

• Outcome capture: pytest --report-jsonl PATH writes one JSON record per test (outcome, wasxfail, concise crash message, record_property metadata) from a pytest_runtest_logreport hook — xdist-safe, no new dependencies. The file is truncated per run and its parent directory created, so a local re-run reports the run you just did rather than merging with the last one. The Google Sheet tests record category, source, and query metadata structurally.
• Report generator (src/babel_validation/tools/generate_report.py): aggregates raw outcomes per test (worst outcome wins across phases and subtests, strict XPASS classified as xpassed), merges each row's metadata across every environment that recorded it, fetches each environment's /status endpoint, and writes report.json (latest run, full detail) plus history.jsonl (one compact summary line per run, for trends). A malformed record, a history line that is not a run object, or a targets.ini section that defines only one of the two service URLs degrades that one value rather than aborting the report.
• Dashboard website (Astro + Vue, replacing the old pages): a status-by-environment table in deployment order (exp → dev → ci → ci-es → test → prod) with difference highlighting; a paginated tests-by-environment matrix defaulting to "interesting only" rows (failing, unexpectedly passing, or differing across environments); a collapsible filter panel whose state — filters, page, page size and the expanded test — lives in the URL and is restored intact by a shared link, with a "Copy link to this view" button; links out to GitHub issues, direct NodeNorm/NameRes queries, and Babel Explorer; and a trends page over the run history.
• Daily workflow (.github/workflows/dashboard.yaml): cron + workflow_dispatch; one 45-minute-capped pytest run per target (a failing suite is normal — the report is the artifact), then generate, build, and deploy to gh-pages. The release-triggered deploy workflow is deleted — it would publish a site with stale or missing data.
• Local development and tests: npm run fetch-data downloads the live site's report.json and history.jsonl into website/public/data/, so frontend work does not need a local suite run first (generating them from scratch is still documented in README.md, "The dashboard website"). npm test runs vitest over the components' URL round-tripping, filtering and pagination — logic the Python suite cannot see — and the Tests workflow runs it alongside the Python unit tests.
• Deleted: website-vue3-vite/ (never deployed) and the Autocomplete/Prefix Comparator pages and their dependencies.

Security

Everything feeding the report is untrusted (issue bodies, sheet cells, service responses) and the site is public. The generator is the choke point: text is repr-escaped and truncated, /status responses pass a key whitelist, issue ids and source URLs become links only when they match the targets.ini Repositories allowlist, blocklist details are withheld entirely, and the Google Sheet IDs come from environment variables and never appear in the output (guarded by a unit test). The Vue components render report values via {{ }} interpolation only and construct links from validated parts — including the test key from the URL, which is looked up with Object.hasOwn so a crafted ?test= cannot pull a member of Object.prototype into the table.

Notes for review

• Includes the two sheet-ID commits also cherry-picked into Read the Google Sheet IDs from the environment instead of checking them in #117; whichever merges second will need a trivial rebase.
• tests/unit/test_generate_report.py covers the generator, including its untrusted-input and degradation paths; website/test/dashboard.test.js covers the components' URL and pagination logic. Both run in CI on every pull request.

Before merge (blocking)

• Remove the temporary push: branches: [replace-website] trigger from dashboard.yaml (added only because workflow_dispatch needs the file on the default branch; pre-merge test runs used it)
• Verify the live deploy from the test run: dashboard renders at https://translatorsri.github.io/babel-validation/ with all six environments populated and the GitHub issue tests reporting results
• Reconcile gh-pages ownership with Publish a daily cross-repository Babel milestones page to GitHub Pages #112 (milestones page): both workflows force-push the whole gh-pages branch, so as written they will overwrite each other's content. The plan is to confirm there is nothing worth saving in Publish a daily cross-repository Babel milestones page to GitHub Pages #112 and close it without merging.

Future work

Everything still open below has been carried into #120's follow-up list — work from that one. This list is kept only as the record of what this PR knew about.

• Remove scala-validation/ and its workflow — filed as Remove the unmaintained scala-validation/ directory and its workflow #119 (milestone: Babel Validation v1.1)
• Re-share the Google Sheets under new IDs and rotate the repository secrets (see Read the Google Sheet IDs from the environment instead of checking them in #117)
• Trend graphs on the trends page, once the table proves insufficient
• Rebase whichever of Read the Google Sheet IDs from the environment instead of checking them in #117 and this PR merges second — the two sheet-ID commits exist in both
• Refresh README's example pytest transcripts (they still show configfile: pytest.ini and old tool versions)

Review history — kept for anyone tracing why a particular line looks the way it does; the durable conclusions are in the code, the tests and CLAUDE.md above.

A code review of the branch found nine issues, fixed in four commits:

• Shared links did not survive (8029211): Vue registers watchers before created(), so the watchers saw readUrl()'s assignments as user edits — the filters handler reset page to 1 and rewrote the URL, and a link carrying ?page=3 always landed on page 1. The pinned-row lookup also indexed report.results raw, so ?test=constructor pinned an Object.prototype member and blanked the table. The trends page's ../ back-link resolved above <base href="/babel-validation/"> to the site root.
• Generator robustness (95b12c6): fetch_status called .rstrip() on a URL targets.ini need not define, read_raw_records passed a non-string id to parse_nodeid, and append_history checked only that prior lines were valid JSON, not that they were run objects — each aborted the whole report. Result annotation also ran only for the first environment seen, so a row whose first environment failed during setup (before record_property runs) lost its category and service links even though the others had recorded them.
• --report-jsonl ergonomics (9de751f): the README's own example died in pytest_configure because the parent directory did not exist, and appending meant a local re-run kept a fixed test red under worst-outcome-wins. CI never noticed either, since it writes one fresh file per target.
• Workflow trigger (ac880f5): removed, as the blocking box above required.

Two items from the future-work list were then pulled into this PR rather than deferred. A run where no target recorded a single test is now dropped from history on the way in and not written on the way out (b23df3b), which retires the all-zero shakedown row on the live site at the next daily run instead of by hand. And the frontend gained a vitest harness (ab55b78): four of its six tests fail against the components as they stood before the review round above.

Earlier rounds of the branch fixed the xdist option-loading trap, the root .gitignore swallowing website/src/lib/, and node-ID key normalization; those traps are written up in CLAUDE.md.

🤖 Generated with Claude Code