Add property-based correctness tests for DJ

[robinld]

Summary

Add a deliberately small, extensible Hypothesis correctness harness for DJ OSS.

• Generate immutable metric, filter, request, and small fact/dimension graph scenarios. Compare DJ-generated metric SQL with separately written DuckDB raw-table reference queries.
• Check metric decomposition/rollup, SQL parse-print semantics, AST printing, and pre-aggregation equivalence. A fixed case requires an eligible pre-aggregation to be used, so routing cannot silently disappear into discarded examples.
• Keep discovered DJ bugs as narrow strict-XFAIL reproductions; distinguish oracle mismatches from API/setup failures and preserve SQL NULL, NaN, and infinity semantics.
• Document local, deterministic CI, and deeper manual nightly profiles. Broader generator families are intentionally follow-up work.

Known-bug parametrizations remain skipped (14 cases), with 9 strict-XFAIL repros that will turn into failures when those bugs are fixed.

Test Plan

• PR has an associated public issue.
• DJ_PBT_PROFILE=ci ../.venv/bin/pytest -q -n 2 --dist=loadscope tests/property --without-integration --without-slow-integration from datajunction-server — 36 passed, 14 skipped, 9 xfailed.
• Pre-commit checks on changed property-test files (including Ruff, formatting, and mypy).
• uv lock --check --offline.
• make check — not run locally.
• make test / full-suite coverage — not run locally; CI will exercise the repository suite.

Deployment Plan

No runtime behavior change. This adds tests, a test-only Hypothesis dependency, and contributor guidance.

[netlify]

✅ Deploy Preview for thriving-cassata-78ae72 canceled.

Name	Link
🔨 Latest commit	637bb2f
🔍 Latest deploy log	https://app.netlify.com/projects/thriving-cassata-78ae72/deploys/6abd771cae82ab000868fadd

[greptile-apps]

Confidence Score: 4/5

[Low risk] Adds property-based test suite with test infrastructure.

The PR appears safe to merge, with non-blocking opportunities to preserve rollup coverage and bound test resource growth.

Findings

1.  Whole metric families skipped ▶
2.  Generated graphs accumulate resources ▶

Summary

This PR adds Hypothesis-based correctness tests for metric rollups, SQL printing, graph queries, and pre-aggregation routing, plus test profiles and contributor guidance.

• Independent DuckDB reference queries check generated metric SQL.
• Fixed strict-XFAIL cases retain reproductions for known bugs.
• The rollup exclusions leave several metric families outside generated coverage, and graph resources accumulate during longer runs.

_{Reviews (1) · Last reviewed commit: "Add property-based correctness tests for..."}

[greptile-apps]

Whole metric families skipped The skip marker removes every generated rollup case for sample variance, sample standard deviation, covariance, and correlation, including inputs that do not trigger the known bugs. The fixed XFAILs cover only narrow examples, so this property cannot catch other regressions in those metric families. Restrict the excluded inputs instead of skipping each metric entirely.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

Generated graphs accumulate resources Each example creates DuckDB tables and API nodes without removing them. The shared fixtures retain those resources across examples, and materialization examples also create pre-aggregation tables. This makes longer runs, especially the tenfold nightly profile, progressively more expensive. Clean up each example’s resources or bound their lifetime.