Skip to content

@W-23545283 native lib benchmarks#137

Open
andres-rad wants to merge 13 commits into
masterfrom
W-23545283-native-lib-benchmarks
Open

@W-23545283 native lib benchmarks#137
andres-rad wants to merge 13 commits into
masterfrom
W-23545283-native-lib-benchmarks

Conversation

@andres-rad

Copy link
Copy Markdown
Contributor

What

Adds a language-agnostic benchmark harness for the DataWeave native-lib Node wrapper under a new top-level benchmarks/ directory. It measures four metrics — cold-start, first-run (compile+exec), warm steady-state, and streaming throughput — emits results in a shared JSON schema, and renders a comparison report. Architected so future Python and Scala-engine runners plug in unchanged.

W-23545283.

How it works

  • benchmarks/corpus/ — the shared contract every runner consumes: manifest.json (each case has an immutable id, a .dwl script, optional inputs, and an explicit metrics[]), 6 scripts, committed inputs, and a deterministic large-input generator.
  • benchmarks/schema/result.schema.json — the JSON schema all runners' output conforms to (env carries weaveVersion/commit/dwlibBuildId for attribution).
  • benchmarks/lib/ — dependency-free helpers: stats (percentiles + MB/s), manifest (load/validate + fail-fast orphan-id check), env (env stamp).
  • benchmarks/runners/node/ — the reference runner: in-process warm/streaming (warm-bench), a fresh-process spawn harness for cold-start/first-run (coldstart), and emit (orchestrate → validate ids → write result JSON).
  • benchmarks/report/report.mjs — joins result files by case id, renders a case×metric table with a Δ-vs-baseline column and a weave-version skew banner; dedupes to the latest result per runner. --emit is a reserved exporter seam.
  • native-lib:benchmark Gradle task — opt-in via -Pbenchmark=true, off by default (never in the normal build/test path).

Design decisions

  • Timing is a self-contained process.hrtime.bigint() sampling harness (not vitest bench) so a future JVM-engine harness can mirror the exact methodology for a fair cross-surface comparison.
  • first-run is measured in the spawn harness (fresh process = genuine cold compilation).
  • Results are local-only/gitignored; durable history (Grafana/exporter) is a deferred follow-up, with env.commit/dwlibBuildId + immutable id + the --emit seam in place so it's a bolt-on.

Testing

  • 19 harness tests pass (node --test across benchmarks/**/*.test.mjs), including integration tests that drive the real native library.
  • Verified the full Gradle path end-to-end on GraalVM (./gradlew native-lib:benchmark -Pbenchmark=true, exit 0) — builds dwlib, runs all four metrics, renders the report. Numbers are reproducible run-over-run (~1–3% variance).

Out of scope (follow-ups)

  • Python runner (runners/python/).
  • The Scala-engine baseline harness — must live in the data-weave repo (depends on DataWeaveScriptingEngine); reads this corpus + emits this schema. Deserves its own spec (JVM cold-start/JIT-warmup methodology).
  • Results datastore / dashboard.

Design spec and implementation plan are included under docs/superpowers/.

@andres-rad
andres-rad requested a review from a team as a code owner July 22, 2026 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant