Skip to content

Add compare mode for base vs PR regression checks - #57

Open
wouterlucas wants to merge 1 commit into
mainfrom
feature/compare-runner
Open

wouterlucas wants to merge 1 commit into
mainfrom
feature/compare-runner

Conversation

@wouterlucas

Copy link
Copy Markdown
Contributor

Compares two already-built dist/ bundles back-to-back on the same machine/browser and fails when the candidate regresses beyond threshold. Intended for infrequent renderer/FTL release/version-bump PRs.

Usage:

node index.js --compare --base renderer-base --current renderer-pr --threshold 10
  • New: testrunner/compareResults.js (pure compare + markdown logic, unit-verified with synthetic data)
  • index.js: --compare --base/--current/--threshold/--strict/--startup-only, reuses the existing runBenchmark/runStartup/processResults pipeline for both variants
  • Default gates on overall avgMean, startup average, and bundle size; per-op regressions reported without failing unless --strict
  • Emits results/compare_vs.json + .md (PR comment/step summary) plus side-by-side HTML; exit 1 on regression
  • npm run compare passthrough, readme docs

Compares two already-built dist/ bundles back-to-back on the same
machine/browser (throughput + memory + startup + bundle size) and exits
non-zero when the candidate regresses beyond --threshold (default 10%).

- testrunner/compareResults.js: pure compare + markdown logic
- index.js: --compare --base/--current/--threshold/--strict/--startup-only
- Emits results/compare_<base>_vs_<current>.json/.md plus side-by-side HTML
- Intended for infrequent renderer/FTL release/version-bump PRs

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant