Skip to content

Add cold-startup benchmark (navigation start to first frame) - #56

Merged
wouterlucas merged 1 commit into
mainfrom
feature/startup-benchmark
Oct 25, 2026
Merged

wouterlucas merged 1 commit into
mainfrom
feature/startup-benchmark

Conversation

@wouterlucas

Copy link
Copy Markdown
Contributor

What

Adds a cold-startup benchmark alongside the existing throughput (Done!) and memory (Memory!) tests: fresh browser context, load the app, render 1,000 rows, wait for the first renderer-idle, stop the clock. Timing is navigation-relative, so JS download/parse/execute, framework init, font atlas load and scene build are all included (bundle size matters).

Covers all variants: Renderer (+canvasfonts), FTL (+canvas/bitmap/webgl-bitmap), Lightning2 (+core), Blits.

How it works

  • Each framework app gets a #startup entry point that renders 1,000 rows once and logs Startup! with the navigation-relative time (Blits gets a /startup route since its router is hash-based)
  • New testrunner/runStartup.js: sequential cold loads, fresh context per run, 6x CPU throttle applied before navigation, per-run timeout, all-fail returns startup: -1
  • Startup is processed with a plain relative mean vs the fastest framework and reported in its own table section — deliberately excluded from avgMean, so historical Overall scores are unchanged
  • writeResults emits a single results_*.html with only the sections that have data, keeping framework columns aligned
  • node index.js --startup-only runs just startup + bundle sizes (STARTUP_RUNS, HEADLESS=true supported)

Validation done

  • Unit: report pipeline with fixtures (avgMean unchanged, failure renders Failed, no unreplaced placeholders)
  • Smoke: Playwright runner against fixture pages (multi-run stats, timeout failure path)
  • Real apps: rebuilt Blits and ran the actual startup flow headless (2/2 cold loads reported)

Not included

No generated output (results/, official_results/) and no build side-effects (package-lock link churn) — run on the default benchmark hardware and publish results from there.

Measures navigation start to first renderer idle after rendering 1,000
rows, so JS download/parse/execute, framework init, font atlas load and
scene build are all included in the timing.

- New testrunner/runStartup.js: sequential cold loads with a fresh browser
  context per run, 6x CPU throttling applied before navigation (unlike the
  throughput benchmark which throttles after page load)
- New #startup entry point in all framework variants (Renderer, FTL,
  Lightning2 incl. canvas/bitmap variants, Blits via a /startup route
  since Blits is hash-routed); each logs a Startup! console signal with
  the navigation-relative time
- Startup results are processed with a plain relative mean against the
  fastest framework and reported in their own table section, deliberately
  excluded from avgMean so historical Overall scores are unchanged
- writeResults renders a single results_*.html with only the sections
  that have data, keeping framework columns aligned
- New --startup-only flag runs just the startup benchmark plus bundle
  sizes (tunable via STARTUP_RUNS, HEADLESS=true for headless Chromium)
@wouterlucas
wouterlucas merged commit 95f03b0 into main Oct 25, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant