From f10cf74c2d59793ba1d88e9e43a112722f2acb7d Mon Sep 17 00:00:00 2001 From: Johnny Wilson Dougherty Date: Thu, 16 Jul 2026 12:34:20 +0530 Subject: [PATCH] chore: repository hygiene and metadata accuracy pass - remove committed AI-assistant working artifacts (.superpowers/, docs/superpowers/ published to the docs site) and git-ignore them - remove dead MANIFEST.in (hatchling ignores it; sdist verified byte-identical without it) - remove the never-executed nested freshdata-benchmarks/.github workflow and correct that README's CI-execution/published-results claims - anchor hatchling sdist include patterns to the repo root so the sdist ships exactly the documented file set - declare least-privilege GITHUB_TOKEN permissions in every workflow - align contributor docs (CONTRIBUTING.md, README.md, QUALITY_OPS.md, docs/contributing.md) with the exact commands CI runs; drop the unenforced ruff-format pre-commit hook and document why - update SECURITY.md supported-versions table to 1.1.x - merge duplicate Unreleased headings in CHANGELOG.md and record this pass - add CITATION.cff and a documentation issue template --- .github/ISSUE_TEMPLATE/documentation.md | 20 + .github/workflows/ci.yml | 5 + .github/workflows/fetch-fixtures.yml | 3 + .github/workflows/nightly-real-model.yml | 3 + .github/workflows/plugin-contract.yml | 3 + .github/workflows/reproducibility.yml | 3 + .github/workflows/wheel-size.yml | 3 + .gitignore | 3 + .pre-commit-config.yaml | 6 +- .superpowers/sdd/task-4-report.md | 110 -- .superpowers/sdd/task-5-report.md | 85 - .superpowers/sdd/task-6-report.md | 202 --- CHANGELOG.md | 89 +- CITATION.cff | 16 + CONTRIBUTING.md | 19 +- MANIFEST.in | 35 - QUALITY_OPS.md | 2 +- README.md | 21 +- RELEASE.md | 4 +- SECURITY.md | 4 +- docs/contributing.md | 11 +- docs/performance-investigation.md | 16 +- ...reshdata-performance-baseline-profiling.md | 1418 ----------------- ...hdata-semantic-performance-optimization.md | 440 ----- .../plans/2026-07-14-freshdata-truthbench.md | 687 -------- ...onfirmed-performance-bottlenecks-design.md | 262 --- ...reshdata-performance-scalability-design.md | 360 ----- .../2026-07-14-freshdata-truthbench-design.md | 498 ------ .../.github/workflows/benchmark.yml | 186 --- freshdata-benchmarks/README.md | 33 +- pyproject.toml | 26 +- 31 files changed, 193 insertions(+), 4380 deletions(-) create mode 100644 .github/ISSUE_TEMPLATE/documentation.md delete mode 100644 .superpowers/sdd/task-4-report.md delete mode 100644 .superpowers/sdd/task-5-report.md delete mode 100644 .superpowers/sdd/task-6-report.md create mode 100644 CITATION.cff delete mode 100644 MANIFEST.in delete mode 100644 docs/superpowers/plans/2026-07-11-freshdata-performance-baseline-profiling.md delete mode 100644 docs/superpowers/plans/2026-07-13-freshdata-semantic-performance-optimization.md delete mode 100644 docs/superpowers/plans/2026-07-14-freshdata-truthbench.md delete mode 100644 docs/superpowers/specs/2026-07-11-freshdata-confirmed-performance-bottlenecks-design.md delete mode 100644 docs/superpowers/specs/2026-07-11-freshdata-performance-scalability-design.md delete mode 100644 docs/superpowers/specs/2026-07-14-freshdata-truthbench-design.md delete mode 100644 freshdata-benchmarks/.github/workflows/benchmark.yml diff --git a/.github/ISSUE_TEMPLATE/documentation.md b/.github/ISSUE_TEMPLATE/documentation.md new file mode 100644 index 0000000..51f7196 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/documentation.md @@ -0,0 +1,20 @@ +--- +name: Documentation issue +about: Report a mistake, gap, or unclear section in the docs, README, or examples +title: '[DOCS] ' +labels: documentation +assignees: '' + +--- + +**Where** +Link to the page or file (e.g. a https://freshcode-org.github.io/freshdata/ URL, +`README.md`, or an `examples/` script) and the section heading. + +**What is wrong or missing** +A clear description of the problem: incorrect statement, code snippet that +doesn't run, broken link, missing explanation, outdated API, etc. + +**Suggested fix (optional)** +If you already know what the text or snippet should say, propose it here — +small doc fixes make great first PRs. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6c721d2..26ef67f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -12,6 +12,11 @@ concurrency: group: ci-${{ github.ref }} cancel-in-progress: true +# Least privilege by default; jobs that need more (nightly alert issue, +# coverage-badge push) declare their own elevated permissions. +permissions: + contents: read + jobs: quality-fast: runs-on: ubuntu-latest diff --git a/.github/workflows/fetch-fixtures.yml b/.github/workflows/fetch-fixtures.yml index e6dea4c..56e5d41 100644 --- a/.github/workflows/fetch-fixtures.yml +++ b/.github/workflows/fetch-fixtures.yml @@ -5,6 +5,9 @@ on: schedule: - cron: "0 6 * * 1" # weekly Monday 06:00 UTC +permissions: + contents: read + jobs: fetch: runs-on: ubuntu-latest diff --git a/.github/workflows/nightly-real-model.yml b/.github/workflows/nightly-real-model.yml index 633a1cb..e725cb3 100644 --- a/.github/workflows/nightly-real-model.yml +++ b/.github/workflows/nightly-real-model.yml @@ -7,6 +7,9 @@ on: - cron: "0 4 * * *" workflow_dispatch: +permissions: + contents: read + jobs: real-model: if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' diff --git a/.github/workflows/plugin-contract.yml b/.github/workflows/plugin-contract.yml index 19eb6c3..0e0c93a 100644 --- a/.github/workflows/plugin-contract.yml +++ b/.github/workflows/plugin-contract.yml @@ -8,6 +8,9 @@ on: push: branches: [main] +permissions: + contents: read + jobs: plugin-contract: runs-on: ubuntu-latest diff --git a/.github/workflows/reproducibility.yml b/.github/workflows/reproducibility.yml index 740b92f..712b13d 100644 --- a/.github/workflows/reproducibility.yml +++ b/.github/workflows/reproducibility.yml @@ -10,6 +10,9 @@ on: push: branches: [main] +permissions: + contents: read + jobs: reproducibility: runs-on: ubuntu-latest diff --git a/.github/workflows/wheel-size.yml b/.github/workflows/wheel-size.yml index f576787..f715cbe 100644 --- a/.github/workflows/wheel-size.yml +++ b/.github/workflows/wheel-size.yml @@ -10,6 +10,9 @@ on: env: MAX_WHEEL_BYTES: "2000000" +permissions: + contents: read + jobs: wheel-guard: runs-on: ubuntu-latest diff --git a/.gitignore b/.gitignore index c397747..474627f 100644 --- a/.gitignore +++ b/.gitignore @@ -15,6 +15,9 @@ htmlcov/ .DS_Store .ipynb_checkpoints/ +# AI-assistant working artifacts (plans, task reports) — never committed +.superpowers/ + # MkDocs build output site/ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index abd53ae..02f9ede 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,4 +1,7 @@ -# Run `pre-commit install` after cloning. CI mirrors these checks. +# Run `pre-commit install` after cloning. The ruff and mypy hooks match the +# required CI lane; the hygiene hooks (whitespace, YAML/TOML syntax) are +# pre-commit-only. There is deliberately no formatter hook: the codebase is +# not `ruff format`-clean, and CI does not enforce formatting. repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v4.6.0 @@ -16,7 +19,6 @@ repos: hooks: - id: ruff args: [--fix] - - id: ruff-format - repo: https://github.com/pre-commit/mirrors-mypy rev: v1.11.2 diff --git a/.superpowers/sdd/task-4-report.md b/.superpowers/sdd/task-4-report.md deleted file mode 100644 index 7418575..0000000 --- a/.superpowers/sdd/task-4-report.md +++ /dev/null @@ -1,110 +0,0 @@ -# Task 4 Report: Add finance, healthcare, retail, and CRM gold datasets - -## Status - -DONE. The four deterministic 16-row domain builders are registered and covered by focused and adjacent TruthBench tests. - -## TDD evidence - -### RED - -```text -PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov -``` - -Observed 16 failures before implementation: each new domain was absent from the fixture registry (`FixtureError: unknown fixture domain`). - -### GREEN - -```text -PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov -``` - -Result: `28 passed`. - -```text -PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py tests/truthbench/test_models_exact.py tests/truthbench/test_schema.py -q --no-cov -``` - -Result: all focused, model, and schema tests green. - -Static checks: - -```text -python -m ruff check benchmarks/truthbench/fixtures tests/truthbench/test_fixtures.py -python -m ruff format --check benchmarks/truthbench/fixtures tests/truthbench/test_fixtures.py -mypy --ignore-missing-imports --explicit-package-bases benchmarks/truthbench/fixtures tests/truthbench/conftest.py tests/truthbench/test_fixtures.py -git diff --check -``` - -All passed. - -## Files - -- `benchmarks/truthbench/fixtures/finance.py` — finance frame and oracle labels for Apple semantic traps, numeric/currency/date defects, protected ticker conflict, and synthetic PII canaries. -- `benchmarks/truthbench/fixtures/healthcare.py` — healthcare frame and oracle labels for ICD/LOINC, temperature and dose units, FHIR/partial/impossible dates, Unicode, and PHI canaries. -- `benchmarks/truthbench/fixtures/retail.py` — retail frame and oracle labels for leading-zero identifiers, free/return quantities, locale currency formats, mojibake/HTML, multilingual values, and review PII. -- `benchmarks/truthbench/fixtures/crm.py` — CRM frame and oracle labels for Unicode/contact ambiguity, lifecycle contradiction, zero-width/hidden PII, Apple semantic traps, and protected IDs. -- `benchmarks/truthbench/fixtures/__init__.py` — registry entries and stable domain order. -- `tests/truthbench/test_fixtures.py` — domain completeness, disposition, deterministic-seed, and adversarial marker tests. - -## Self-review - -- Every builder emits exactly 16 rows with stable string indexes, fixed `2026-01-15` UTC reference metadata, explicit locale metadata, complete physical-cell labels, and at least 12 injected adversarial cells. -- All four dispositions are represented per domain. Row cases cover exact duplicates and removed rows; schema cases cover added, removed, renamed, reordered, and type-drifted columns without mislabeling absent cells. -- Sensitive values are whole-value synthetic `.invalid`, `TB-*`, or `555-01xx` forms, preserving the base builder's redaction and canary invariants. Zero-width examples are intentionally non-sensitive representation traps. -- Seed values are included only in a deterministic batch marker; same-seed builds produce byte-stable serialized oracle payloads and identical fixture hashes for seeds `1729` and `2718`. -- No FreshData runtime, LLM/provider, network, or external data dependency is used. - -## Concerns - -- The legacy `minimal` fixture remains registered for backwards compatibility; the four Task 4 domains are appended in stable order. A later registry task may choose to retire `minimal` once its callers migrate. -- Row/schema expectations are metadata cases (the physical frame remains rectangular), consistent with the oracle contract that removed rows/columns cannot have cell labels. - -## Review follow-up: healthcare reference codes and content assertions - -### RED - -After review, focused content tests were strengthened to inspect each actual `GoldCell.family`, disposition, and adversarial frame value. The first run exposed four failures: the healthcare rare-code test rejected the placeholder `G rare`; finance, retail, and CRM injection-count assertions correctly counted only non-preserve dispositions rather than all injected cells. - -```text -PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov -``` - -Result before fixes: `4 failed, 24 passed`. - -### GREEN - -Healthcare preserve cases now use values present in the bundled reference sets (`Z79.4`, `F17.210`, and `9843-4`), and tests load those references plus validate ICD/LOINC syntax. Content tests assert the finance, healthcare, retail, and CRM families directly against physical cells; injected-cell counts use non-background families, including preserve injections. - -```text -PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov -``` - -Result: `28 passed`. - -Complete adjacent verification (`tests/truthbench/test_fixtures.py`, `tests/truthbench/test_models_exact.py`, and `tests/truthbench/test_schema.py`) passed. Ruff check/format, mypy, and `git diff --check` also passed. - -## Review follow-up: contract-gap family coverage - -### RED - -Added direct assertions for the finance USD/EUR/INR conflict and zero-width memo, healthcare protected-DOB repair conflict and MRN tail canary, and exact row/schema family sets for finance, healthcare, and CRM. The initial run exposed the missing `zero-width-memo` family label (the value existed but was grouped under the broader invisible-PII family). - -```text -PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov -``` - -Result: `1 failed, 30 passed` (`StopIteration` while locating the required zero-width memo family). - -### GREEN - -The zero-width memo cell now has its own `zero-width-memo` family; all assertions inspect actual frame values, dispositions, sensitivity, and exact case-family sets. - -```text -PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov -``` - -Result: `31 passed`. - -Complete fixture/models/schema verification passed (`... passed`), as did Ruff check/format, mypy, and `git diff --check`. diff --git a/.superpowers/sdd/task-5-report.md b/.superpowers/sdd/task-5-report.md deleted file mode 100644 index 6ad9f93..0000000 --- a/.superpowers/sdd/task-5-report.md +++ /dev/null @@ -1,85 +0,0 @@ -# Task 5 Report: Complete eight-domain TruthBench corpus - -## Status - -DONE. Logistics, government, education, and insurance now have deterministic 16-row gold fixtures. The registry is the stable alphabetical eight-domain order (`crm`, `education`, `finance`, `government`, `healthcare`, `insurance`, `logistics`, `retail`) with the temporary minimal registry removed. - -## TDD evidence - -### RED - -After adding domain-content tests, the focused fixture suite failed with the expected missing-builder errors for all four new domains (`FixtureError: unknown fixture domain`). The failure run was: - -```text -PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov -``` - -### GREEN - -The focused suite now passes: - -```text -PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov -``` - -Result: `47 passed`. - -Adjacent fixture/model/schema verification: - -```text -PYTHONPATH=src python -m pytest \ - tests/truthbench/test_fixtures.py \ - tests/truthbench/test_models_exact.py \ - tests/truthbench/test_schema.py -q --no-cov -``` - -Result: `112 passed`. - -Static checks: - -```text -python -m ruff check benchmarks/truthbench/fixtures tests/truthbench/test_fixtures.py -python -m ruff format --check benchmarks/truthbench/fixtures tests/truthbench/test_fixtures.py -mypy --ignore-missing-imports --explicit-package-bases \ - benchmarks/truthbench/fixtures tests/truthbench/test_fixtures.py -git diff --check -``` - -All passed. - -## Coverage - -- `logistics.py` covers valid UN/LOCODE-like references, kg/lb and C/F units, cross-timezone windows, 24:00 transport values, address PII, late tracking, and protected shipment IDs. -- `government.py` covers leading-zero IDs, Indian/international grouping, fiscal/calendar ambiguity, multilingual labels, restricted national IDs, mixed legacy encoding, and retention/repair policy contradiction. -- `education.py` covers student IDs, letter/percentage/GPA scales, school-year ambiguity, zero scores, enrollment ordering, guardian contacts, FERPA notes, and protected grade-policy conflict. -- `insurance.py` covers policy/claim IDs, premium/reserve currency mismatch, negative reserve review, incident/report ordering, state contradiction, claimant/medical PII, and protected policy numbers. -- All four builders emit complete physical-cell labels, all four dispositions, row duplicate/removal cases, five schema drift cases, fixed UTC/reference metadata, deterministic seed batches, and privacy-safe synthetic canaries. -- Corpus-level tests assert all required trap categories occur across the eight domains. - -## Concerns - -None. No FreshData runtime, LLM/provider, network, or external data dependency is used. - -## Review follow-up: explicit contract coverage - -### RED - -The new required-family contract test intentionally used the repaired numeric output (`95`) as the adversarial frame value for education `edu-07`. The focused test failed because the actual frame value is the required raw value `"95%"` while the repair oracle separately stores `95.0` as its expected output. - -```text -PYTHONPATH=src python -m pytest \ - tests/truthbench/test_fixtures.py::test_required_domain_families_match_actual_values_and_dispositions \ - -q --no-cov -``` - -Observed: one failure at `edu-07` (`'95%' != 95`). - -### GREEN - -The contract now asserts the raw value, family, disposition, and typed repair output. It also uses an explicit per-domain family mapping covering every required category (including logistics lb/F units, government IDs/grouping/language/fiscal/protected case, education scales/contacts/protected grade, and insurance IDs/grouped premium/PII/protected policy). - -```text -PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov -``` - -Result: `48 passed`. diff --git a/.superpowers/sdd/task-6-report.md b/.superpowers/sdd/task-6-report.md deleted file mode 100644 index ae4f228..0000000 --- a/.superpowers/sdd/task-6-report.md +++ /dev/null @@ -1,202 +0,0 @@ -# Task 6 Report: Privacy-safe values and exhaustive sink scanning - -## Status - -DONE. `SinkScanner` now scans normalized canary variants across nested TruthBench -sinks and emits only `Leak(canary_id, variant, path)` metadata. Redaction markers -contain run-scoped HMAC-SHA256 digests and never matched text. - -## TDD evidence - -### RED - -```text -PYTHONPATH=src python -m pytest tests/truthbench/test_privacy.py -q --no-cov -``` - -Observed collection failure before implementation: - -```text -ModuleNotFoundError: No module named 'benchmarks.truthbench.privacy' -``` - -### GREEN - -Focused privacy suite: - -```text -PYTHONPATH=src python -m pytest tests/truthbench/test_privacy.py -q --no-cov -``` - -Result: `18 passed`. - -Adjacent TruthBench suites: - -```text -PYTHONPATH=src python -m pytest \ - tests/truthbench/test_privacy.py \ - tests/truthbench/test_fixtures.py \ - tests/truthbench/test_models_exact.py \ - tests/truthbench/test_schema.py -q --no-cov -``` - -Result: `131 passed`. - -Static checks: - -```text -python -m ruff check benchmarks/truthbench/privacy.py \ - benchmarks/truthbench/__init__.py tests/truthbench/test_privacy.py -python -m ruff format --check benchmarks/truthbench/privacy.py \ - benchmarks/truthbench/__init__.py tests/truthbench/test_privacy.py -mypy --ignore-missing-imports --explicit-package-bases \ - benchmarks/truthbench/privacy.py tests/truthbench/test_privacy.py -git diff --check -``` - -## Review follow-up: MultiIndex and hostile-label privacy hardening - -### RED - -Added regression tests for MultiIndex columns/index level names, tuple-label -structure during redaction, and custom non-string labels whose stringification -contains a canary. Before the fix: - -```text -pytest -q tests/truthbench/test_privacy.py --no-cov -``` - -Result: `24 passed, 2 failed`. - -The failures were the expected defects: redacting tuple labels converted them -to lists and raised pandas `ValueError` (length mismatch), while hostile custom -labels were not scanned. - -### GREEN - -Structure-preserving label traversal/redaction now handles tuple/MultiIndex -levels and names, and sanitizes custom label paths before reporting or replacing -them with digest markers: - -```text -pytest -q tests/truthbench/test_privacy.py --no-cov -``` - -Result: `26 passed`. - -Adjacent TruthBench verification: - -```text -pytest -q tests/truthbench --no-cov -``` - -Result: `139 passed`. - -Static checks: - -```text -ruff check benchmarks/truthbench/privacy.py tests/truthbench/test_privacy.py -ruff format --check benchmarks/truthbench/privacy.py tests/truthbench/test_privacy.py -mypy benchmarks/truthbench/privacy.py -git diff --check -``` - -All passed. - -All passed. - -## Files - -- `benchmarks/truthbench/privacy.py` — `Leak`, `PrivacySafeValue`, named - normalizers, run-scoped HMAC scanner, recursive redaction, typed redaction - handling, pandas/dataclass/bytes support, self-test, and named sink entry points. -- `benchmarks/truthbench/__init__.py` — exports privacy scanner primitives. -- `tests/truthbench/test_privacy.py` — mutation matrix for every required - normalized form, nested sink coverage, redaction/self-test behavior, and typed - redaction digest safety. - -## Self-review - -- Leak objects contain only identifiers, transform labels, and JSONPath-like - locations; `repr(leaks)` cannot repeat canary text. -- Literal, case-folded, whitespace-stripped, punctuation-stripped, digit-only, - URL-decoded, HTML-unescaped, UTF-8/hex/escape bytes, NFKC/NFC/NFD, - zero-width-removed, and JSON-escaped forms are covered. -- Mapping, sequence, dataclass, pandas DataFrame/Series/Index, exception, - report, plan, generated-code, stream, markup, JSON, and failure-artifact sinks - are traversed. Exact redacted `TypedValue` payloads are treated as safe and - their HMAC digests are not re-scanned as plaintext. -- Redaction is recursive, emits `[REDACTED:]`, and `self_test` raises on - an unredacted result while accepting the scanner's own redacted output. - -## Concerns - -- The scanner intentionally treats digit-only normalized forms conservatively; - very short numeric canaries can match unrelated text. Fixtures use synthetic - identifiers and email/phone canaries, so this does not affect the bundled - corpus. -- Redacting a pandas object may change a sensitive column to object/string dtype, - which is preferable to retaining a raw value in an audit sink. - -## Follow-up RED/GREEN evidence - -A follow-up regression test used the exact sensitive `TypedValue` redaction shape -with a one-digit canary. Before the redacted-payload guard, the digest's hex text -was incorrectly reported as a digit-only leak. After adding the guard: - -```text -PYTHONPATH=src python -m pytest \ - tests/truthbench/test_privacy.py::test_scanner_accepts_exact_typed_redaction_without_scanning_digest \ - -q --no-cov -``` - -Result: `1 passed`. - -## Review follow-up: marker, key, and label hardening - -### RED - -The review regression matrix was run before the hardening changes: - -```text -PYTHONPATH=src python -m pytest tests/truthbench/test_privacy.py -q --no-cov -``` - -Result: `4 failed, 17 passed` for forged redaction markers, sensitive mapping -keys, pandas labels, and the digest compatibility alias. - -### GREEN - -After validating markers against the scanner's own 64-hex HMAC digest set, -redacting mapping keys and pandas column/index/name labels, scanning arbitrary -key stringifications, and adding `digest` as an alias: - -```text -PYTHONPATH=src python -m pytest tests/truthbench/test_privacy.py -q --no-cov -``` - -Result: `23 passed`. - -Adjacent TruthBench verification: - -```text -PYTHONPATH=src python -m pytest \ - tests/truthbench/test_privacy.py \ - tests/truthbench/test_fixtures.py \ - tests/truthbench/test_models_exact.py \ - tests/truthbench/test_schema.py -q --no-cov -``` - -Result: `136 passed`. - -Static checks were rerun after the follow-up and remained clean: - -```text -python -m ruff check benchmarks/truthbench/privacy.py \ - tests/truthbench/test_privacy.py -python -m ruff format --check benchmarks/truthbench/privacy.py \ - tests/truthbench/test_privacy.py -mypy --ignore-missing-imports --explicit-package-bases \ - benchmarks/truthbench/privacy.py tests/truthbench/test_privacy.py -git diff --check -``` diff --git a/CHANGELOG.md b/CHANGELOG.md index f5f2886..58b4354 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,54 @@ adheres to [Semantic Versioning](https://semver.org/). no longer silently accepted — it gets a warning-severity issue with the canonical form as `suggestion` and action `accept_with_warning` (gauntlet finding). +- `docs/trust-claims.md` (claim-to-evidence map for every trust-relevant + README/docs claim, superset of the machine-enforced `CLAIM_REGISTRY`) and + `docs/threat-model.md` (trust boundaries, per-privacy-mode guarantees, + ranked residual risks), both linked in the docs nav. +- `benchmarks/bench_outofcore.py`: subprocess-isolated peak-RSS evidence for + the four engine/output-format combinations on a generated parquet fixture + (per-scenario `ru_maxrss`, wall time, and the `materialized` flag). +- **AI Copilot (experimental)** — `freshdata.experimental.ai_copilot.analyze_dataset`: + deterministic, fully offline dataset analysis that returns a ranked problem + list (PII, policy violations, duplicates, missing values, mixed date + formats, near-duplicate category spellings), a PII warning, an ordered + explainable cleaning plan, and copy-ready freshdata code generated for the + analyzed dataset. Privacy-first: raw string values never enter the report's + `model_context` (every string-like sample column is hash-masked, numeric + values pass through as-is, or samples are omitted entirely with + `privacy="schema_only"`); the payload is SHA-256 fingerprinted in the + audit. An optional `provider` hook (plain `Callable[[str], str]`) allows + plugging in an LLM later — no built-in provider ships, no API key is + needed, and provider failures never break the deterministic report. +- **Flagship demo**: `examples/freshdata_ai_copilot_demo.py` plus the bundled + `examples/data/messy_customers.csv` — the full messy-to-audit-ready story + (analyze → mask → clean under a compiled policy → merge category variants → + re-score trust), and a new docs guide (`docs/ai-copilot.md`). +- `CITATION.cff` so the project can be cited from GitHub's "Cite this + repository" button, and a documentation issue template alongside the + existing bug/feature templates. + +### Changed +- Lint now covers the whole repository (`ruff check .` in CI, closing #54): + benchmark and notebook lint debt fixed, dead code removed + (`harness_metrics` unused gold-labels block), and the ASV-managed + `freshdata-benchmarks/` sub-project excluded as tool-generated. No + runtime behavior changes. +- All CI workflows now declare least-privilege `GITHUB_TOKEN` permissions + (read-only by default; the nightly-alert and coverage-badge jobs keep their + scoped write grants). +- Contributor docs (`CONTRIBUTING.md`, `README.md`, `QUALITY_OPS.md`) now + quote the exact commands CI runs; the pre-commit config no longer ships a + `ruff-format` hook the codebase and CI never enforced. + +### Removed +- Dead packaging/CI config: `MANIFEST.in` (ignored by the hatchling build + backend — the sdist is shaped by `pyproject.toml`) and + `freshdata-benchmarks/.github/workflows/` (nested workflow directories are + never executed by GitHub Actions). +- Committed AI-assistant working artifacts (`.superpowers/`, now + git-ignored) and internal planning notes that were being published to the + documentation site (`docs/superpowers/`). ### Fixed - **Unparseable values are quarantined, never fabricated** (gauntlet finding, @@ -115,38 +163,15 @@ adheres to [Semantic Versioning](https://semver.org/). streaming CLI keep byte-exact output by default and gain an explicit opt-in (`sanitize_formulas=True` / `--sanitize-formulas`) covering the cleaned output and the quarantine export. JSONL/Parquet are never altered. - -### Added -- `docs/trust-claims.md` (claim-to-evidence map for every trust-relevant - README/docs claim, superset of the machine-enforced `CLAIM_REGISTRY`) and - `docs/threat-model.md` (trust boundaries, per-privacy-mode guarantees, - ranked residual risks), both linked in the docs nav. -- `benchmarks/bench_outofcore.py`: subprocess-isolated peak-RSS evidence for - the four engine/output-format combinations on a generated parquet fixture - (per-scenario `ru_maxrss`, wall time, and the `materialized` flag). -- **AI Copilot (experimental)** — `freshdata.experimental.ai_copilot.analyze_dataset`: - deterministic, fully offline dataset analysis that returns a ranked problem - list (PII, policy violations, duplicates, missing values, mixed date - formats, near-duplicate category spellings), a PII warning, an ordered - explainable cleaning plan, and copy-ready freshdata code generated for the - analyzed dataset. Privacy-first: raw string values never enter the report's - `model_context` (every string-like sample column is hash-masked, numeric - values pass through as-is, or samples are omitted entirely with - `privacy="schema_only"`); the payload is SHA-256 fingerprinted in the - audit. An optional `provider` hook (plain `Callable[[str], str]`) allows - plugging in an LLM later — no built-in provider ships, no API key is - needed, and provider failures never break the deterministic report. -- **Flagship demo**: `examples/freshdata_ai_copilot_demo.py` plus the bundled - `examples/data/messy_customers.csv` — the full messy-to-audit-ready story - (analyze → mask → clean under a compiled policy → merge category variants → - re-score trust), and a new docs guide (`docs/ai-copilot.md`). - -### Changed -- Lint now covers the whole repository (`ruff check .` in CI, closing #54): - benchmark and notebook lint debt fixed, dead code removed - (`harness_metrics` unused gold-labels block), and the ASV-managed - `freshdata-benchmarks/` sub-project excluded as tool-generated. No - runtime behavior changes. +- `SECURITY.md` supported-versions table updated to the current 1.1.x line. +- Source distribution now contains exactly the documented file set: the + hatchling `include` patterns are anchored to the repo root, so unanchored + names no longer pull in stray matches at any depth (`docs/examples/*.html`, + nested `README.md`s). +- `freshdata-benchmarks/README.md` no longer claims CI execution or + published results the repository never produced; it now documents the ASV + suite as a locally-run comparative benchmark, separate from the CleanBench + CI workflow. ## [1.1.1] - 2026-07-06 diff --git a/CITATION.cff b/CITATION.cff new file mode 100644 index 0000000..99898d3 --- /dev/null +++ b/CITATION.cff @@ -0,0 +1,16 @@ +cff-version: 1.2.0 +message: "If you use freshdata in your research or data pipelines, please cite it as below." +title: "freshdata: explainable, decision-preserving data cleaning for pandas" +type: software +authors: + - family-names: "Dougherty" + given-names: "Johnny Wilson" +license: MIT +repository-code: "https://github.com/FreshCode-Org/freshdata" +url: "https://freshcode-org.github.io/freshdata/" +keywords: + - data-cleaning + - data-quality + - pandas + - preprocessing + - machine-learning diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 112e87c..d74b905 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -22,17 +22,28 @@ philosophy are very welcome. git clone https://github.com/FreshCode-Org/freshdata cd freshdata python -m venv .venv && source .venv/bin/activate -pip install -e ".[dev]" +pip install -e ".[dev,ml]" +pre-commit install # optional: run the same hooks CI expects on every commit ``` ## Checks to run before a PR +These mirror the required CI lane: + ```bash -pytest # all tests -ruff check src tests # lint -mypy # types +pytest -m "not online and not large" # fast test lane (CI-required; includes the 93% coverage gate) +ruff check . # lint +mypy src/freshdata # types ``` +Bare `pytest` additionally collects the `online`/`large` marked tests, which +need network access and local datasets — they run in the nightly CI lane, not +on PRs. If you change user-facing behavior, update the matching `docs/` page +and add a note under `Unreleased` in `CHANGELOG.md`. + +First time contributing? See the +[first-PR walkthrough](docs/community/first-pr.md). + ## Adding an online dataset fixture 1. Add an entry to [`tests/fixtures/online/registry.json`](tests/fixtures/online/registry.json) diff --git a/MANIFEST.in b/MANIFEST.in deleted file mode 100644 index cafddb6..0000000 --- a/MANIFEST.in +++ /dev/null @@ -1,35 +0,0 @@ -# Source distribution contents for freshdata. -# The wheel is built from src/freshdata; this file shapes the sdist tarball. - -include README.md -include CHANGELOG.md -include LICENSE -include SECURITY.md -include CONTRIBUTING.md -include CONTRIBUTING_DOMAINS.md -include CODE_OF_CONDUCT.md -include RELEASE.md -include pyproject.toml -include py.typed - -# Typed marker and any package data shipped inside the import package. -recursive-include src/freshdata py.typed *.json *.yaml - -# Tests are useful for downstream packagers reproducing the suite. -recursive-include tests *.py *.json *.csv - -# Examples help users get started from the sdist. -recursive-include examples *.py *.md - -# Keep build noise out of the tarball. -exclude .pre-commit-config.yaml -prune docs -prune site -prune notebooks -prune benchmarks -prune scripts -prune packaging -global-exclude *.pyc *.pyo __pycache__ .DS_Store -global-exclude .coverage *.so -prune .github -prune .git diff --git a/QUALITY_OPS.md b/QUALITY_OPS.md index 3fd720a..52c1a1c 100644 --- a/QUALITY_OPS.md +++ b/QUALITY_OPS.md @@ -38,7 +38,7 @@ Track these weekly (rolling 4-week trend): Required lane locally: ```bash -ruff check src tests +ruff check . mypy src/freshdata pytest -m "not online and not large" ``` diff --git a/README.md b/README.md index ad71c53..60e3220 100644 --- a/README.md +++ b/README.md @@ -52,8 +52,9 @@ every new dataset and want an audit trail they can hand to a reviewer. removes outliers blindly. - **pandas-first, Polars-optional** — pandas + NumPy core; pass a Polars frame and get one back, with optional Polars/DuckDB/Spark execution backends for - scaling beyond pandas (see `docs/backends.md` for the measured - out-of-core boundary per backend and output format). + scaling beyond pandas (see the + [backends guide](https://freshcode-org.github.io/freshdata/backends/) for the + measured out-of-core boundary per backend and output format). - **CLI included** — `clean`, `plan`, `apply-plan`, `profile`, `learn`, and `trust` subcommands for scripting and CI pipelines without writing Python. - **Typed, tested, fast** — fully type-hinted (`py.typed`), vectorized, with a @@ -201,12 +202,14 @@ See [`examples/README.md`](https://github.com/FreshCode-Org/freshdata/blob/main/ ``` freshdata/ -├── src/freshdata/ # library source (engine, domains, enterprise, execution backends, CLI) -├── tests/ # pytest suite -├── examples/ # runnable usage examples -├── docs/ # mkdocs-material documentation site -├── benchmarks/ # CleanBench accuracy/performance benchmark harness -└── crates/ # optional Rust acceleration crate (freshcore) +├── src/freshdata/ # library source (engine, domains, enterprise, execution backends, CLI) +├── tests/ # pytest suite +├── examples/ # runnable usage examples +├── docs/ # mkdocs-material documentation site +├── benchmarks/ # CleanBench accuracy/performance benchmark harness +├── freshdata-benchmarks/ # comparative ASV benchmark suite (run locally) +├── training/ # dev-only training pipeline for the semantic models +└── crates/ # optional Rust acceleration crate (freshcore) ``` ## CLI reference @@ -239,7 +242,7 @@ python -m venv .venv && source .venv/bin/activate pip install -e ".[dev,ml]" pytest -m "not online and not large" # fast lane, matches CI -ruff check src tests # lint +ruff check . # lint, matches CI mypy src/freshdata # typecheck ``` diff --git a/RELEASE.md b/RELEASE.md index aa8fcf5..d6dd445 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -49,8 +49,8 @@ Push a version tag; the `Release` workflow runs a quality gate first then builds and publishes via PyPI **Trusted Publishing** (OIDC, no stored token): ```bash -git tag v0.5.0 -git push origin v0.5.0 +git tag vX.Y.Z # must match the version in pyproject.toml / __version__ +git push origin vX.Y.Z ``` One-time setup: add a trusted publisher at diff --git a/SECURITY.md b/SECURITY.md index 4c12af0..f96ccb8 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -10,8 +10,8 @@ backport fixes to older releases — please upgrade to the latest. | Version | Supported | | ------- | ------------------ | -| 1.0.x | :white_check_mark: | -| < 1.0 | :x: | +| 1.1.x | :white_check_mark: | +| < 1.1 | :x: | ## Reporting a Vulnerability diff --git a/docs/contributing.md b/docs/contributing.md index 1e387a6..fec1bb6 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -26,13 +26,14 @@ pre-commit install ## Run the checks ```bash -pytest # full test suite (coverage gate ≥ 93%) -ruff check src tests # lint -ruff format --check src tests -mypy src/freshdata # type check +pytest -m "not online and not large" # fast test lane (coverage gate ≥ 93%) +ruff check . # lint +mypy src/freshdata # type check ``` -All four must pass; CI runs them on Python 3.9–3.13. +All three must pass — they are exactly what the required CI lane runs. The +test matrix repeats the fast lane on Python 3.9–3.13; `online`/`large` marked +tests need network access and run in the nightly lane instead. ## Build the docs locally diff --git a/docs/performance-investigation.md b/docs/performance-investigation.md index f3ac60f..8512ed3 100644 --- a/docs/performance-investigation.md +++ b/docs/performance-investigation.md @@ -339,7 +339,7 @@ remains byte-for-byte unchanged relative to `6f6c2fe`. The final acceptance environment was macOS 15.5 arm64, Python 3.12.13, pandas 2.3.3, and NumPy 2.4.6. At verification the only worktree dirt was the pre-existing untracked `.venv-qa/` directory; no raw benchmark output or -`.superpowers/sdd` report is part of the production change. +assistant working artifact is part of the production change. ## 12. Peak-memory comparison @@ -437,11 +437,11 @@ Final Task 9.5 gate results: | Full pytest suite | 0 | 3,238 passed, 7 skipped, 18 warnings in 223.55s; no failures | The focused and full suites were run serially. The final Task 9.5 -documentation build's Material-for-MkDocs upstream notice and four existing -`docs/superpowers/` pages reported as not in `nav` were informational and did -not fail strict mode. The earlier documentation verification run reported two -such un-navigated pages; those were likewise informational and not a -strict-build error. +documentation build's Material-for-MkDocs upstream notice and a few +internal planning pages reported as not in `nav` (since removed from the +docs tree) were informational and did not fail strict mode. The earlier +documentation verification run reported two such un-navigated pages; those +were likewise informational and not a strict-build error. ```bash .venv-qa/bin/python -m pytest tests/performance -q --no-cov @@ -460,8 +460,8 @@ Results from the earlier documentation verification run: | Production-source identity diff | 0 | No output; `src/freshdata` remains byte-for-byte unchanged from `6f6c2fe` | The documentation build also printed Material for MkDocs' upstream MkDocs 2.0 -notice and INFO that two `docs/superpowers/` pages are not in `nav`; neither was -a strict-build error. +notice and INFO that two internal planning pages (since removed from the docs +tree) were not in `nav`; neither was a strict-build error. ### Optimization-specific acceptance verification diff --git a/docs/superpowers/plans/2026-07-11-freshdata-performance-baseline-profiling.md b/docs/superpowers/plans/2026-07-11-freshdata-performance-baseline-profiling.md deleted file mode 100644 index 08f6aee..0000000 --- a/docs/superpowers/plans/2026-07-11-freshdata-performance-baseline-profiling.md +++ /dev/null @@ -1,1418 +0,0 @@ -# FreshData Performance Baseline and Profiling Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Build the deterministic benchmark and profiling system, capture the immutable `6f6c2fe` baseline across the required scale matrix, and produce authoritative evidence that decides which production optimizations deserve separate implementation plans. - -**Architecture:** A development-only `benchmarks/performance/` package generates deterministic mixed frames, executes one benchmark case per subprocess, records schema-validated JSON, profiles functions/allocations/observed pandas operations, and renders Markdown comparisons. This phase does not change the cleaning implementation: it establishes the architecture summary, baseline, confirmed bottlenecks, and rejected hypotheses that gate later TDD optimization plans. - -**Tech Stack:** Python 3.9+, pandas 1.5-2.x, NumPy, psutil from the existing `bench` extra, standard-library `argparse`, `cProfile`, `pstats`, `subprocess`, `statistics`, `threading`, `tracemalloc`, pytest, jsonschema, GitHub Actions, MkDocs. - -## Global Constraints - -- Python support remains `>=3.9`, including the project's tested Python 3.9-3.13 matrix. -- pandas support remains `>=1.5,<3`; NumPy remains `>=1.21`. -- The default cleaning strategy remains `balanced`, with the current representation-repair defaults unchanged. -- Public function and class signatures, configuration fields, return types, warnings, exceptions, index semantics, and audit/report contracts remain compatible. -- No unrelated runtime dependency is introduced. Profiling additions use the standard library and already-declared benchmark/development dependencies. -- Baseline source is commit `6f6c2fe`; the design-only commit `21bec88` is not a production-code change. -- One warm-up and five measured repetitions are the default for reportable results. -- A claimed win must be at least 10%, exceed twice observed run-to-run variability, and introduce no meaningful primary-workload regression. -- Preserve the user's untracked `.venv-qa/`; never stage, edit, or remove it. -- Run this plan in an isolated workspace created by `superpowers:using-git-worktrees` before Task 1. - -## Scope Decomposition and Evidence Gate - -This is Phase 1 of the approved design. It intentionally ends before production optimization because exact optimization code cannot be specified responsibly until profiles identify the functions and lines responsible. The remaining work is split as follows: - -1. **Phase 1 — this plan:** benchmark infrastructure, immutable baseline, profiling, architecture summary, root-cause decisions. -2. **Phase 2 — evidence-derived plans:** one TDD plan per confirmed pandas bottleneck; rejected hypotheses receive no production change. -3. **Phase 3 — scale and backend plan:** before/after matrix, native/materialization evaluation, scheduled workflow, documentation corrections. -4. **Phase 4 — verification plan:** complete compatibility matrix, all repository gates, and the 17-section final report audit. - -Phase 2 plans may be written only after Task 8 produces `confirmed_root_causes` entries with exact files, lines, stage percentages, copy/scan evidence, and affected benchmark cases. - -## File Structure - -- `benchmarks/performance/__init__.py` — supported row/width constants and public benchmark-tool exports. -- `benchmarks/performance/models.py` — immutable case/environment/result data models and JSON conversion. -- `benchmarks/performance/datasets.py` — deterministic mixed-schema DataFrame generator. -- `benchmarks/performance/schema.py` — JSON Schema and semantic result validation. -- `benchmarks/performance/environment.py` — package, Git, hardware, and platform capture. -- `benchmarks/performance/memory.py` — peak-RSS sampler used only by benchmark workers. -- `benchmarks/performance/worker.py` — execute one case in a fresh process and write one result. -- `benchmarks/performance/runner.py` — expand matrices and supervise timeout-safe subprocess workers. -- `benchmarks/performance/instrumentation.py` — cProfile, tracemalloc, copy-call, conversion, and scan observations. -- `benchmarks/performance/baselines.py` — comparable component-level pandas operations. -- `benchmarks/performance/analysis.py` — variability, ratios, stage aggregation, and hypothesis classification. -- `benchmarks/performance/render.py` — deterministic Markdown report generation. -- `benchmarks/performance/cli.py` / `__main__.py` — `run`, `profile`, `analyze`, and `render` commands. -- `tests/performance/` — deterministic unit/contract tests for every module and small end-to-end cases. -- `benchmarks/results/performance/` — ignored raw run directories plus explicitly committed compact evidence snapshots. -- `docs/performance-investigation.md` — architecture, commands, baseline tables, profiles, confirmed causes, and rejected hypotheses; later phases extend it to all 17 deliverables. -- `.github/workflows/performance-large.yml` — manual/scheduled large matrix. -- `Makefile` — local commands for CI-sized and large benchmark profiles. -- `.gitignore` — ignore raw results while allowing named compact evidence snapshots. - ---- - -### Task 1: Deterministic Mixed-Schema Dataset Generator - -**Files:** -- Create: `benchmarks/performance/__init__.py` -- Create: `benchmarks/performance/datasets.py` -- Create: `tests/performance/conftest.py` -- Create: `tests/performance/test_datasets.py` - -**Interfaces:** -- Consumes: NumPy and pandas only. -- Produces: `WIDTHS: dict[str, int]`, `DATASET_TYPES: tuple[str, ...]`, - `DatasetSpec`, and `make_mixed_frame(spec: DatasetSpec) -> pd.DataFrame`. - -- [ ] **Step 1: Write failing generator tests** - -```python -# tests/performance/test_datasets.py -from __future__ import annotations - -import pandas as pd -import pytest - -from benchmarks.performance.datasets import DatasetSpec, WIDTHS, make_mixed_frame - - -@pytest.mark.parametrize("width, n_cols", WIDTHS.items()) -def test_mixed_frame_has_exact_shape_and_required_roles(width: str, n_cols: int) -> None: - df = make_mixed_frame(DatasetSpec(rows=1_000, width=width, seed=42)) - assert df.shape == (1_000, n_cols) - assert {"record_id", "target", "numeric_0", "category_0", "text_0", "event_time_0"} <= set(df.columns) - assert df["target"].isna().sum() > 0 - assert df.duplicated().sum() > 0 - assert pd.api.types.is_categorical_dtype(df["category_0"].dtype) - assert isinstance(df["event_time_0"].dtype, pd.DatetimeTZDtype) - - -def test_mixed_frame_is_deterministic_and_seed_sensitive() -> None: - spec = DatasetSpec(rows=2_000, width="medium", seed=7) - assert make_mixed_frame(spec).equals(make_mixed_frame(spec)) - assert not make_mixed_frame(spec).equals( - make_mixed_frame(DatasetSpec(rows=2_000, width="medium", seed=8)) - ) - - -def test_mixed_frame_covers_nullable_outlier_and_high_cardinality_cases() -> None: - df = make_mixed_frame(DatasetSpec(rows=5_000, width="medium", seed=42)) - assert str(df["nullable_int_0"].dtype) == "Int64" - assert df["nullable_int_0"].isna().any() - assert df["numeric_0"].isna().any() - assert df["numeric_0"].max() > 1_000 - assert df["high_cardinality_0"].nunique() > 2_000 - - -@pytest.mark.parametrize( - "dataset_type, expected_prefix", - [ - ("numeric", "numeric_"), ("categorical", "category_"), - ("string", "text_"), ("nullable", "nullable_int_"), - ("datetime", "datetime_"), - ("high_cardinality", "high_cardinality_"), - ], -) -def test_family_profile_dominates_non_role_columns(dataset_type: str, expected_prefix: str) -> None: - df = make_mixed_frame( - DatasetSpec(rows=1_000, width="medium", seed=42, dataset_type=dataset_type) - ) - family_columns = [name for name in df.columns if name.startswith(expected_prefix)] - assert len(family_columns) >= 24 -``` - -- [ ] **Step 2: Run the tests and verify the import failure** - -Run: `python -m pytest tests/performance/test_datasets.py -q --no-cov` - -Expected: FAIL during collection because `benchmarks.performance.datasets` does not exist. - -- [ ] **Step 3: Implement the generator** - -```python -# benchmarks/performance/datasets.py -from __future__ import annotations - -from dataclasses import dataclass - -import numpy as np -import pandas as pd - -WIDTHS = {"narrow": 8, "medium": 32, "wide": 128} -DATASET_TYPES = ( - "mixed", "numeric", "categorical", "string", "nullable", "datetime", - "high_cardinality", -) - - -@dataclass(frozen=True) -class DatasetSpec: - rows: int - width: str - seed: int = 42 - dataset_type: str = "mixed" - - def __post_init__(self) -> None: - if self.rows < 1: - raise ValueError("rows must be >= 1") - if self.width not in WIDTHS: - raise ValueError(f"width must be one of {sorted(WIDTHS)}") - if self.dataset_type not in DATASET_TYPES: - raise ValueError(f"dataset_type must be one of {DATASET_TYPES}") - - -def _nullable_int(rng: np.random.Generator, rows: int) -> pd.Series: - values = pd.array(rng.integers(0, 10_000, rows), dtype="Int64") - values[rng.random(rows) < 0.10] = pd.NA - return pd.Series(values) - - -def make_mixed_frame(spec: DatasetSpec) -> pd.DataFrame: - rng = np.random.default_rng(spec.seed) - rows = spec.rows - numeric = rng.normal(100.0, 20.0, rows) - numeric[rng.random(rows) < 0.10] = np.nan - numeric[rng.choice(rows, max(1, rows // 100), replace=False)] *= 20.0 - categories = pd.Categorical( - rng.choice(["alpha", "beta", "gamma", None], rows, p=[0.35, 0.3, 0.25, 0.1]) - ) - frame = pd.DataFrame( - { - "record_id": np.arange(rows, dtype=np.int64), - "target": pd.array(rng.choice([0, 1, None], rows, p=[0.47, 0.48, 0.05]), dtype="Int8"), - "numeric_0": numeric, - "nullable_int_0": _nullable_int(rng, rows), - "category_0": categories, - "text_0": pd.array([f"free form note {i % 97}" if i % 11 else None for i in range(rows)], dtype="string"), - "event_time_0": pd.date_range("2024-01-01", periods=rows, freq="min", tz="UTC"), - "high_cardinality_0": pd.array([f"key-{spec.seed}-{i}" for i in range(rows)], dtype="string"), - } - ) - factories = ( - lambda i: pd.Series(rng.normal(i, 1.0, rows), name=f"numeric_{i}"), - lambda i: _nullable_int(rng, rows).rename(f"nullable_int_{i}"), - lambda i: pd.Series(pd.Categorical(rng.choice(["a", "b", "c", None], rows)), name=f"category_{i}"), - lambda i: pd.Series(pd.array([f"value {i}-{j % 211}" for j in range(rows)], dtype="string"), name=f"text_{i}"), - lambda i: pd.Series(pd.date_range("2020-01-01", periods=rows, freq="h"), name=f"datetime_{i}"), - lambda i: pd.Series(pd.array([f"hc-{i}-{j}" for j in range(rows)], dtype="string"), name=f"high_cardinality_{i}"), - ) - family_index = { - "numeric": 0, "nullable": 1, "categorical": 2, "string": 3, - "datetime": 4, "high_cardinality": 5, - }.get(spec.dataset_type) - if family_index is not None: - frame = frame[["record_id", "target"]].copy() - i = 1 - while frame.shape[1] < WIDTHS[spec.width]: - index = family_index if family_index is not None else (i - 1) % len(factories) - series = factories[index](i) - frame[series.name] = series - i += 1 - if rows >= 100: - duplicate_count = max(1, rows // 100) - frame = pd.concat( - [frame.iloc[:-duplicate_count], frame.iloc[:duplicate_count].copy()], - ignore_index=True, - ) - return frame -``` - -```python -# benchmarks/performance/__init__.py -from .datasets import DATASET_TYPES, DatasetSpec, WIDTHS, make_mixed_frame - -__all__ = ["DATASET_TYPES", "DatasetSpec", "WIDTHS", "make_mixed_frame"] -``` - -```python -# tests/performance/conftest.py -from __future__ import annotations - -# Performance-tool modules live in the repository and are imported as the -# namespace package ``benchmarks.performance``. No sys.path mutation is needed -# when tests run with ``python -m pytest`` from the repository root. -``` - -- [ ] **Step 4: Run generator tests** - -Run: `python -m pytest tests/performance/test_datasets.py -q --no-cov` - -Expected: PASS for all width, determinism, dtype, defect, and role assertions. - -- [ ] **Step 5: Commit the generator** - -```bash -git add benchmarks/performance/__init__.py benchmarks/performance/datasets.py tests/performance/conftest.py tests/performance/test_datasets.py -git commit -m "bench: add deterministic scalability datasets" -``` - -### Task 2: Typed Cases, Environment Capture, and Result Schema - -**Files:** -- Create: `benchmarks/performance/models.py` -- Create: `benchmarks/performance/environment.py` -- Create: `benchmarks/performance/schema.py` -- Create: `tests/performance/test_models_schema.py` - -**Interfaces:** -- Consumes: `DatasetSpec` from Task 1. -- Produces: `BenchmarkCase`, `EnvironmentInfo`, `BenchmarkResult`, `capture_environment()`, `RESULT_SCHEMA`, and `validate_result(payload)`. - -- [ ] **Step 1: Write failing model/schema tests** - -```python -# tests/performance/test_models_schema.py -from __future__ import annotations - -from dataclasses import replace - -import pytest - -from benchmarks.performance.environment import capture_environment -from benchmarks.performance.models import BenchmarkCase, BenchmarkResult -from benchmarks.performance.schema import validate_result - - -def test_case_id_is_stable_and_configuration_sensitive() -> None: - case = BenchmarkCase(rows=10_000, width="narrow", config_name="default", options={}) - assert case.case_id == BenchmarkCase(rows=10_000, width="narrow", config_name="default", options={}).case_id - assert case.case_id != replace(case, return_report=True).case_id - - -def test_environment_contains_required_reproduction_fields() -> None: - env = capture_environment() - assert env.python_version - assert env.pandas_version - assert env.numpy_version - assert env.git_commit - assert env.platform - assert env.cpu_count_logical >= 1 - - -def test_result_round_trip_validates() -> None: - case = BenchmarkCase(rows=10_000, width="narrow", config_name="default", options={}) - result = BenchmarkResult.completed( - case=case, - environment=capture_environment(), - samples_seconds=[1.0, 1.1, 0.9], - peak_rss_bytes=1_000_000, - peak_python_bytes=500_000, - input_bytes=250_000, - command="python -m benchmarks.performance worker", - ) - payload = result.to_dict() - validate_result(payload) - assert BenchmarkResult.from_dict(payload).case == case - - -def test_completed_result_rejects_empty_samples() -> None: - with pytest.raises(ValueError, match="samples_seconds"): - BenchmarkResult.completed( - case=BenchmarkCase(rows=10_000, width="narrow", config_name="default", options={}), - environment=capture_environment(), - samples_seconds=[], - peak_rss_bytes=0, - peak_python_bytes=0, - input_bytes=1, - command="x", - ) -``` - -- [ ] **Step 2: Verify the tests fail because the modules are absent** - -Run: `python -m pytest tests/performance/test_models_schema.py -q --no-cov` - -Expected: FAIL during collection for missing `models`, `environment`, or `schema`. - -- [ ] **Step 3: Implement exact case and result models** - -```python -# benchmarks/performance/models.py -from __future__ import annotations - -import hashlib -import json -import statistics -from dataclasses import asdict, dataclass, field -from typing import Any - - -@dataclass(frozen=True) -class BenchmarkCase: - rows: int - width: str - config_name: str - options: dict[str, Any] - dataset_type: str = "mixed" - return_report: bool = False - backend: str = "pandas" - output_format: str = "pandas" - seed: int = 42 - warmups: int = 1 - repetitions: int = 5 - - @property - def case_id(self) -> str: - raw = json.dumps(asdict(self), sort_keys=True, separators=(",", ":"), default=str) - return hashlib.sha256(raw.encode()).hexdigest()[:16] - - -@dataclass(frozen=True) -class EnvironmentInfo: - git_commit: str - git_dirty: bool - python_version: str - pandas_version: str - numpy_version: str - freshdata_version: str - optional_versions: dict[str, str | None] - platform: str - processor: str - cpu_count_logical: int - cpu_count_physical: int | None - total_ram_bytes: int | None - - -@dataclass(frozen=True) -class BenchmarkResult: - schema_version: int - status: str - case: BenchmarkCase - environment: EnvironmentInfo - samples_seconds: list[float] = field(default_factory=list) - median_seconds: float | None = None - min_seconds: float | None = None - max_seconds: float | None = None - stdev_seconds: float | None = None - coefficient_of_variation: float | None = None - throughput_rows_per_second: float | None = None - peak_rss_bytes: int | None = None - peak_python_bytes: int | None = None - input_bytes: int | None = None - input_to_peak_ratio: float | None = None - command: str = "" - error_type: str | None = None - error_message: str | None = None - output_fingerprint: str | None = None - report_fingerprint: str | None = None - result_type: str | None = None - - @classmethod - def completed(cls, *, case: BenchmarkCase, environment: EnvironmentInfo, - samples_seconds: list[float], peak_rss_bytes: int, - peak_python_bytes: int, input_bytes: int, command: str, - output_fingerprint: str | None = None, - report_fingerprint: str | None = None, - result_type: str | None = None) -> "BenchmarkResult": - if not samples_seconds: - raise ValueError("samples_seconds must not be empty") - median = statistics.median(samples_seconds) - stdev = statistics.stdev(samples_seconds) if len(samples_seconds) > 1 else 0.0 - return cls( - schema_version=1, status="completed", case=case, environment=environment, - samples_seconds=samples_seconds, median_seconds=median, - min_seconds=min(samples_seconds), max_seconds=max(samples_seconds), - stdev_seconds=stdev, - coefficient_of_variation=(stdev / median) if median else 0.0, - throughput_rows_per_second=(case.rows / median) if median else None, - peak_rss_bytes=peak_rss_bytes, peak_python_bytes=peak_python_bytes, - input_bytes=input_bytes, - input_to_peak_ratio=(peak_rss_bytes / input_bytes) if input_bytes else None, - command=command, output_fingerprint=output_fingerprint, - report_fingerprint=report_fingerprint, result_type=result_type, - ) - - def to_dict(self) -> dict[str, Any]: - return asdict(self) - - @classmethod - def from_dict(cls, payload: dict[str, Any]) -> "BenchmarkResult": - data = dict(payload) - data["case"] = BenchmarkCase(**data["case"]) - data["environment"] = EnvironmentInfo(**data["environment"]) - return cls(**data) -``` - -- [ ] **Step 4: Implement environment capture and schema validation** - -```python -# benchmarks/performance/environment.py -from __future__ import annotations - -import importlib.metadata -import os -import platform -import subprocess - -import numpy as np -import pandas as pd - -import freshdata - -from .models import EnvironmentInfo - - -def _version(name: str) -> str | None: - try: - return importlib.metadata.version(name) - except importlib.metadata.PackageNotFoundError: - return None - - -def capture_environment() -> EnvironmentInfo: - commit = subprocess.run(["git", "rev-parse", "HEAD"], check=True, capture_output=True, text=True).stdout.strip() - dirty = bool(subprocess.run(["git", "status", "--porcelain"], check=True, capture_output=True, text=True).stdout) - try: - import psutil - physical = psutil.cpu_count(logical=False) - total_ram = int(psutil.virtual_memory().total) - except ImportError: - physical = None - total_ram = None - return EnvironmentInfo( - git_commit=commit, - git_dirty=dirty, - python_version=platform.python_version(), - pandas_version=pd.__version__, - numpy_version=np.__version__, - freshdata_version=freshdata.__version__, - optional_versions={name: _version(name) for name in ("polars", "duckdb", "pyspark", "pyarrow")}, - platform=platform.platform(), processor=platform.processor(), - cpu_count_logical=os.cpu_count() or 1, - cpu_count_physical=physical, total_ram_bytes=total_ram, - ) -``` - -```python -# benchmarks/performance/schema.py -from __future__ import annotations - -from typing import Any - -RESULT_SCHEMA = { - "$schema": "https://json-schema.org/draft/2020-12/schema", - "type": "object", - "required": ["schema_version", "status", "case", "environment", "samples_seconds", "command"], - "properties": { - "schema_version": {"const": 1}, - "status": {"enum": ["completed", "failed", "timeout", "oom", "skipped"]}, - "case": {"type": "object", "required": ["rows", "width", "config_name", "options", "dataset_type", "return_report", "backend", "output_format", "seed", "warmups", "repetitions"]}, - "environment": {"type": "object", "required": ["git_commit", "python_version", "pandas_version", "numpy_version", "freshdata_version", "platform"]}, - "samples_seconds": {"type": "array", "items": {"type": "number", "minimum": 0}}, - "command": {"type": "string"}, - }, -} - - -def validate_result(payload: dict[str, Any]) -> None: - import jsonschema - jsonschema.validate(payload, RESULT_SCHEMA) - if payload["status"] == "completed": - if not payload["samples_seconds"]: - raise ValueError("completed result requires samples_seconds") - for field in ("median_seconds", "min_seconds", "max_seconds", "peak_rss_bytes", "peak_python_bytes", "input_bytes"): - if payload.get(field) is None: - raise ValueError(f"completed result requires {field}") -``` - -- [ ] **Step 5: Run and pass model/schema tests** - -Run: `python -m pytest tests/performance/test_models_schema.py -q --no-cov` - -Expected: PASS. - -- [ ] **Step 6: Commit the data contracts** - -```bash -git add benchmarks/performance/models.py benchmarks/performance/environment.py benchmarks/performance/schema.py tests/performance/test_models_schema.py -git commit -m "bench: define performance result contract" -``` - -### Task 3: Isolated Worker Timing and Peak-Memory Measurement - -**Files:** -- Create: `benchmarks/performance/memory.py` -- Create: `benchmarks/performance/worker.py` -- Create: `tests/performance/test_worker.py` - -**Interfaces:** -- Consumes: `BenchmarkCase`, `BenchmarkResult`, `DatasetSpec`, `capture_environment()`. -- Produces: `PeakRSS`, `execute_case(case: BenchmarkCase, command: str) -> BenchmarkResult`, and `worker_main(case_path, result_path)`. - -- [ ] **Step 1: Write failing worker tests** - -```python -# tests/performance/test_worker.py -from __future__ import annotations - -import pandas as pd - -from benchmarks.performance.models import BenchmarkCase -from benchmarks.performance.worker import execute_case - - -def test_worker_records_warm_samples_memory_and_embedded_report() -> None: - case = BenchmarkCase( - rows=500, width="narrow", config_name="default", options={"verbose": False}, - return_report=False, warmups=1, repetitions=2, - ) - result = execute_case(case, command="unit-test") - assert result.status == "completed" - assert len(result.samples_seconds) == 2 - assert result.peak_rss_bytes is not None and result.peak_rss_bytes >= 0 - assert result.peak_python_bytes is not None and result.peak_python_bytes > 0 - assert result.input_bytes is not None and result.input_bytes > 0 - - -def test_worker_report_flag_does_not_change_cleaned_values() -> None: - base = BenchmarkCase(rows=500, width="narrow", config_name="default", options={"verbose": False}, warmups=0, repetitions=1) - without_report = execute_case(base, command="unit-test") - with_report = execute_case( - BenchmarkCase(**{**base.__dict__, "return_report": True}), command="unit-test" - ) - assert without_report.status == with_report.status == "completed" - assert without_report.output_fingerprint == with_report.output_fingerprint - assert without_report.report_fingerprint == with_report.report_fingerprint - assert without_report.result_type == "CleanResult" - assert with_report.result_type == "tuple[CleanResult,CleanReport]" -``` - -- [ ] **Step 2: Verify the worker tests fail** - -Run: `python -m pytest tests/performance/test_worker.py -q --no-cov` - -Expected: FAIL because `worker.py` and `memory.py` do not exist. - -- [ ] **Step 3: Implement the RSS sampler** - -```python -# benchmarks/performance/memory.py -from __future__ import annotations - -import gc -import threading - - -class PeakRSS: - def __init__(self, interval_seconds: float = 0.01) -> None: - import psutil - self._process = psutil.Process() - self._interval = interval_seconds - self._stop = threading.Event() - self._thread: threading.Thread | None = None - self._baseline = 0 - self._peak = 0 - - def __enter__(self) -> "PeakRSS": - gc.collect() - self._baseline = self._process.memory_info().rss - self._peak = self._baseline - self._thread = threading.Thread(target=self._sample, daemon=True) - self._thread.start() - return self - - def _sample(self) -> None: - while not self._stop.wait(self._interval): - self._peak = max(self._peak, self._process.memory_info().rss) - - def __exit__(self, *_args: object) -> None: - self._peak = max(self._peak, self._process.memory_info().rss) - self._stop.set() - if self._thread is not None: - self._thread.join() - - @property - def increase_bytes(self) -> int: - return max(0, self._peak - self._baseline) -``` - -- [ ] **Step 4: Implement one-case execution** - -```python -# benchmarks/performance/worker.py -from __future__ import annotations - -import gc -import json -import time -import tracemalloc -from pathlib import Path - -import freshdata as fd - -from .datasets import DatasetSpec, make_mixed_frame -from .environment import capture_environment -from .memory import PeakRSS -from .models import BenchmarkCase, BenchmarkResult -from .schema import validate_result - - -def _run_clean(frame, case: BenchmarkCase): - return fd.clean( - frame, config=case.options, return_report=case.return_report, - engine=case.backend, output_format=case.output_format, - ) - - -def _fingerprints(result, return_report: bool) -> tuple[str, str, str]: - import hashlib - import json - import pandas as pd - - if return_report: - cleaned, report = result - result_type = "tuple[CleanResult,CleanReport]" - else: - cleaned, report = result, result.report() - result_type = "CleanResult" - frame_hash = hashlib.sha256() - frame_hash.update(pd.util.hash_pandas_object(cleaned, index=True).values.tobytes()) - frame_hash.update(repr(list(cleaned.columns)).encode()) - frame_hash.update(repr([str(dtype) for dtype in cleaned.dtypes]).encode()) - report_payload = report.to_dict() - report_payload.pop("duration_seconds", None) - report_payload.pop("stage_timings", None) - report_hash = hashlib.sha256( - json.dumps(report_payload, sort_keys=True, default=str).encode() - ).hexdigest() - return frame_hash.hexdigest(), report_hash, result_type - - -def execute_case(case: BenchmarkCase, *, command: str) -> BenchmarkResult: - frame = make_mixed_frame( - DatasetSpec(case.rows, case.width, case.seed, case.dataset_type) - ) - input_bytes = int(frame.memory_usage(index=True, deep=True).sum()) - for _ in range(case.warmups): - _run_clean(frame, case) - samples: list[float] = [] - for _ in range(case.repetitions): - gc.collect() - started = time.perf_counter() - _run_clean(frame, case) - samples.append(time.perf_counter() - started) - gc.collect() - tracemalloc.start() - with PeakRSS() as rss: - measured_result = _run_clean(frame, case) - _current, python_peak = tracemalloc.get_traced_memory() - tracemalloc.stop() - output_fingerprint, report_fingerprint, result_type = _fingerprints( - measured_result, case.return_report - ) - return BenchmarkResult.completed( - case=case, environment=capture_environment(), samples_seconds=samples, - peak_rss_bytes=rss.increase_bytes, peak_python_bytes=python_peak, - input_bytes=input_bytes, command=command, - output_fingerprint=output_fingerprint, - report_fingerprint=report_fingerprint, result_type=result_type, - ) - - -def worker_main(case_path: str, result_path: str, command: str) -> None: - case = BenchmarkCase(**json.loads(Path(case_path).read_text())) - result = execute_case(case, command=command) - payload = result.to_dict() - validate_result(payload) - Path(result_path).write_text(json.dumps(payload, indent=2, sort_keys=True)) -``` - -- [ ] **Step 5: Run worker tests** - -Run: `python -m pytest tests/performance/test_worker.py -q --no-cov` - -Expected: PASS. - -- [ ] **Step 6: Commit worker measurement** - -```bash -git add benchmarks/performance/memory.py benchmarks/performance/worker.py tests/performance/test_worker.py -git commit -m "bench: measure isolated runtime and peak memory" -``` - -### Task 4: Matrix Expansion, Timeout-Safe Subprocess Runner, and CLI - -**Files:** -- Create: `benchmarks/performance/runner.py` -- Create: `benchmarks/performance/cli.py` -- Create: `benchmarks/performance/__main__.py` -- Create: `tests/performance/test_runner_cli.py` - -**Interfaces:** -- Consumes: `BenchmarkCase`, `worker_main`, and JSON result validation. -- Produces: `expand_cases(...)`, `run_case_subprocess(...)`, `run_matrix(...)`, and `python -m benchmarks.performance run`. - -- [ ] **Step 1: Write failing runner/CLI tests** - -```python -# tests/performance/test_runner_cli.py -from __future__ import annotations - -import json - -from benchmarks.performance.cli import main -from benchmarks.performance.runner import expand_cases - - -def test_expand_cases_crosses_required_dimensions() -> None: - cases = expand_cases( - rows=[10_000, 100_000], widths=["narrow", "wide"], - dataset_types=["mixed"], - configs=["default", "conservative"], report_modes=[False, True], - backends=["pandas"], output_formats=["pandas"], seed=42, - warmups=1, repetitions=5, - ) - assert len(cases) == 16 - assert len({case.case_id for case in cases}) == 16 - - -def test_cli_small_matrix_writes_valid_result(tmp_path) -> None: - exit_code = main([ - "run", "--rows", "250", "--widths", "narrow", "--configs", "default", - "--report-modes", "false", "--repetitions", "1", "--warmups", "0", - "--timeout", "60", "--output", str(tmp_path), - ]) - assert exit_code == 0 - payloads = [json.loads(path.read_text()) for path in tmp_path.glob("*.json")] - assert len(payloads) == 1 - assert payloads[0]["status"] == "completed" -``` - -- [ ] **Step 2: Verify runner/CLI tests fail** - -Run: `python -m pytest tests/performance/test_runner_cli.py -q --no-cov` - -Expected: FAIL because `runner.py` and `cli.py` do not exist. - -- [ ] **Step 3: Implement matrix expansion and failure records** - -Implement these exact configuration profiles in `benchmarks/performance/runner.py`: - -```python -CONFIGS = { - "default": {"verbose": False}, - "conservative": {"strategy": "conservative", "verbose": False}, - "representation_off": { - "strategy": "conservative", "column_names": False, - "strip_whitespace": False, "normalize_sentinels": False, - "fix_dtypes": False, "drop_duplicates": False, "verbose": False, - }, - "statistical_off": { - "strategy": "conservative", "impute": None, "outliers": None, - "verbose": False, - }, - "explicit": { - "strategy": "conservative", "impute": "median", "outliers": "flag", - "verbose": False, - }, - "aggressive": {"strategy": "aggressive", "verbose": False}, - "semantic": {"semantic_mode": "assist", "verbose": False}, - "missforest": { - "strategy": "conservative", "impute": "missforest", "verbose": False, - }, -} -``` - -Use `itertools.product` across rows, widths, dataset types, configs, report -modes, backends, and output formats to construct `BenchmarkCase` objects. -`run_case_subprocess` must write the case JSON into a temporary directory, -invoke: - -```python -[ - sys.executable, "-c", - "from benchmarks.performance.worker import worker_main; " - "worker_main(__import__('sys').argv[1], __import__('sys').argv[2], __import__('sys').argv[3])", - str(case_path), str(result_path), command, -] -``` - -with `subprocess.run(..., timeout=timeout_seconds, capture_output=True, text=True)`. On timeout, non-zero exit, missing result, or a result containing `MemoryError`, write a schema-valid `BenchmarkResult` with status `timeout`, `failed`, or `oom`, plus the exception type/message and exact command. Do not raise and lose the remaining matrix. - -- [ ] **Step 4: Implement the CLI parser** - -`benchmarks/performance/cli.py` must parse comma-separated values with these defaults: - -```python -run.add_argument("--rows", default="10000,100000,500000,1000000") -run.add_argument("--widths", default="narrow,medium,wide") -run.add_argument("--dataset-types", default="mixed") -run.add_argument("--configs", default="default,conservative,representation_off,statistical_off,explicit") -run.add_argument("--report-modes", default="false,true") -run.add_argument("--backends", default="pandas") -run.add_argument("--output-formats", default="pandas") -run.add_argument("--seed", type=int, default=42) -run.add_argument("--warmups", type=int, default=1) -run.add_argument("--repetitions", type=int, default=5) -run.add_argument("--timeout", type=int, default=1800) -run.add_argument("--output", required=True) -``` - -The command prints one line per case and ends with counts by status. It returns `0` only when every requested case is completed or explicitly skipped; failures, timeouts, and OOM records return `1` after all cases finish. - -```python -# benchmarks/performance/__main__.py -from .cli import main - -raise SystemExit(main()) -``` - -- [ ] **Step 5: Run runner/CLI tests** - -Run: `python -m pytest tests/performance/test_runner_cli.py -q --no-cov` - -Expected: PASS, with one completed small-case JSON file. - -- [ ] **Step 6: Commit the matrix runner** - -```bash -git add benchmarks/performance/runner.py benchmarks/performance/cli.py benchmarks/performance/__main__.py tests/performance/test_runner_cli.py -git commit -m "bench: add configurable subprocess matrix runner" -``` - -### Task 5: Function, Allocation, Copy, Conversion, and Scan Profiling - -**Files:** -- Create: `benchmarks/performance/instrumentation.py` -- Create: `tests/performance/test_instrumentation.py` -- Modify: `benchmarks/performance/worker.py` -- Modify: `benchmarks/performance/models.py` -- Modify: `benchmarks/performance/schema.py` -- Modify: `benchmarks/performance/cli.py` - -**Interfaces:** -- Consumes: one `BenchmarkCase` and the Task 3 worker execution path. -- Produces: `OperationCounter`, `profile_case(case) -> ProfileResult`, exact function/file/line records, allocation records, stage aggregates, and `python -m benchmarks.performance profile`. - -- [ ] **Step 1: Write failing instrumentation tests** - -```python -# tests/performance/test_instrumentation.py -from __future__ import annotations - -from benchmarks.performance.instrumentation import OperationCounter, profile_case -from benchmarks.performance.models import BenchmarkCase - - -def test_operation_counter_observes_copy_scan_and_conversion_calls() -> None: - import pandas as pd - frame = pd.DataFrame({"a": [1, None, 3], "b": ["1", "2", "3"]}) - with OperationCounter() as counter: - frame.copy(deep=False) - frame["a"].isna() - frame["a"].nunique(dropna=True) - frame["b"].astype("string") - assert counter.counts["dataframe.copy"] == 1 - assert counter.counts["series.isna"] >= 1 - assert counter.counts["series.nunique"] == 1 - assert counter.counts["series.astype"] == 1 - - -def test_profile_case_reports_exact_hot_functions_and_allocations() -> None: - result = profile_case(BenchmarkCase( - rows=500, width="narrow", config_name="default", - options={"verbose": False}, warmups=0, repetitions=1, - )) - assert result.functions - assert all({"file", "line", "function", "self_seconds", "cumulative_seconds", "calls"} <= set(item) for item in result.functions) - assert result.allocations - assert all({"file", "line", "bytes", "count"} <= set(item) for item in result.allocations) - assert result.stages["total"] > 0 - assert result.operations["dataframe.copy"] >= 1 -``` - -- [ ] **Step 2: Verify instrumentation tests fail** - -Run: `python -m pytest tests/performance/test_instrumentation.py -q --no-cov` - -Expected: FAIL because `instrumentation.py` does not exist. - -- [ ] **Step 3: Implement controlled operation counting** - -Create `OperationCounter` as a context manager using `unittest.mock.patch.object` around the original methods. Store originals before patching and route each wrapper directly to its original to avoid recursion. Count these keys: - -```python -METHODS = { - "dataframe.copy": (pd.DataFrame, "copy"), - "series.copy": (pd.Series, "copy"), - "series.isna": (pd.Series, "isna"), - "series.notna": (pd.Series, "notna"), - "series.nunique": (pd.Series, "nunique"), - "series.value_counts": (pd.Series, "value_counts"), - "series.astype": (pd.Series, "astype"), - "dataframe.astype": (pd.DataFrame, "astype"), - "dataframe.corr": (pd.DataFrame, "corr"), - "dataframe.corrwith": (pd.DataFrame, "corrwith"), -} -``` - -Mark these as observed Python method calls, not physical-buffer-copy counts. - -- [ ] **Step 4: Implement cProfile and tracemalloc extraction** - -Add this immutable result contract to `benchmarks/performance/instrumentation.py`: - -```python -from dataclasses import dataclass - - -@dataclass(frozen=True) -class ProfileResult: - functions: list[dict[str, object]] - allocations: list[dict[str, object]] - stages: dict[str, float] - operations: dict[str, int] -``` - -`profile_case(case: BenchmarkCase) -> ProfileResult` must: - -1. Generate the deterministic frame outside the measured profile. -2. Start `tracemalloc`, `cProfile.Profile`, and `OperationCounter`. -3. Execute one `fd.clean` call. -4. Disable profiling and collect the top 100 `pstats.Stats.stats` entries sorted by cumulative time. -5. Collect the top 100 `tracemalloc` line statistics limited to paths containing `/freshdata/` or `/benchmarks/performance/`. -6. Aggregate self time into these stages using normalized path/function rules: - -```python -STAGE_RULES = { - "context": ("engine/context.py",), - "engine_cache": ("engine/cache.py",), - "correlation": ("numeric_corr_matrix", "corr", "corrwith"), - "missing": ("engine/missing.py", "steps/missing.py"), - "outliers": ("engine/outliers.py", "steps/outliers.py"), - "role_inference": ("infer_role", "build_context"), - "dtype_repair": ("steps/dtypes.py",), - "duplicates": ("steps/duplicates.py",), - "audit_events": ("report.py", "CleanReport.add"), - "report_finalization": ("cleaner.py", "memory_bytes"), - "semantic_ml": ("semantic/", "imputation/missforest.py", "sklearn/"), - "backend_conversion": ("adapters/", "execution/backends/"), -} -``` - -Use exclusive self time for percentage totals so nested functions are not double-counted. Preserve cumulative time separately for locating call chains. - -- [ ] **Step 5: Extend result schema and CLI** - -Add optional `profile` to `BenchmarkResult` containing `functions`, -`allocations`, `stages`, and `operations`. Add a `profile` subcommand accepting -the same case selectors as `run` but requiring exactly one row count, width, -config, report mode, backend, and output format. It writes the case's stable -16-character ID followed by `.profile.json` and validates it with the same -schema. - -- [ ] **Step 6: Run instrumentation and focused worker tests** - -Run: `python -m pytest tests/performance/test_instrumentation.py tests/performance/test_worker.py -q --no-cov` - -Expected: PASS. - -- [ ] **Step 7: Commit profiling instrumentation** - -```bash -git add benchmarks/performance/instrumentation.py benchmarks/performance/worker.py benchmarks/performance/models.py benchmarks/performance/schema.py benchmarks/performance/cli.py tests/performance/test_instrumentation.py -git commit -m "bench: profile stages allocations and pandas operations" -``` - -### Task 6: Comparable Pandas Baselines, Analysis, and Markdown Rendering - -**Files:** -- Create: `benchmarks/performance/baselines.py` -- Create: `benchmarks/performance/analysis.py` -- Create: `benchmarks/performance/render.py` -- Create: `tests/performance/test_analysis_render.py` -- Modify: `benchmarks/performance/cli.py` - -**Interfaces:** -- Consumes: completed benchmark/profile JSON files. -- Produces: `measure_pandas_baseline(case, baseline_name)`, a `baseline` CLI - subcommand, component-level pandas baseline samples, slowdown/improvement - ratios, variability-aware comparisons, hypothesis classifications, and - deterministic Markdown. - -- [ ] **Step 1: Write failing comparison/render tests** - -```python -# tests/performance/test_analysis_render.py -from __future__ import annotations - -from benchmarks.performance.analysis import classify_change, classify_hypotheses -from benchmarks.performance.render import render_report - - -def test_change_requires_ten_percent_and_twice_variability() -> None: - assert classify_change(1.0, 0.85, baseline_cv=0.02, candidate_cv=0.02) == "improved" - assert classify_change(1.0, 0.94, baseline_cv=0.01, candidate_cv=0.01) == "noise" - assert classify_change(1.0, 1.12, baseline_cv=0.02, candidate_cv=0.02) == "regressed" - - -def test_hypothesis_classifier_requires_exact_profile_evidence() -> None: - profile = { - "stages": {"correlation": 0.40, "total": 1.0}, - "operations": {"dataframe.corr": 1, "dataframe.copy": 3}, - "functions": [{"file": "src/freshdata/engine/context.py", "line": 207, "function": "numeric_corr_matrix", "self_seconds": 0.4, "cumulative_seconds": 0.4, "calls": 1}], - } - decisions = classify_hypotheses(profile) - assert decisions["unnecessary_correlation"]["status"] == "candidate" - assert decisions["unnecessary_correlation"]["evidence"][0]["line"] == 207 - - -def test_renderer_is_deterministic_and_contains_required_baseline_sections() -> None: - payload = {"environment": {"python_version": "3.12", "pandas_version": "2.3"}, "results": [], "hypotheses": {}} - first = render_report(payload) - assert first == render_report(payload) - for heading in ("Architecture and execution flow", "Reproduction commands", "Baseline benchmark table", "Profiling findings", "Confirmed root causes", "Rejected hypotheses"): - assert f"## {heading}" in first -``` - -- [ ] **Step 2: Verify comparison/render tests fail** - -Run: `python -m pytest tests/performance/test_analysis_render.py -q --no-cov` - -Expected: FAIL because the analysis and render modules do not exist. - -- [ ] **Step 3: Implement semantically comparable pandas baselines** - -Provide named component baselines only: - -```python -BASELINES = { - "shallow_copy": lambda df: df.copy(deep=False), - "numeric_median_fill": lambda df: df.assign(**{ - col: df[col].fillna(df[col].median()) - for col in df.select_dtypes(include="number").columns - if df[col].notna().any() - }), - "duplicates": lambda df: df.drop_duplicates(), - "null_counts": lambda df: df.isna().sum(), -} -``` - -Do not define a pandas baseline for FreshData's full balanced decision/audit pipeline. Label every baseline with its exact operation and include it only when the selected case has matching semantics. - -`measure_pandas_baseline` uses the same one-warm-up/five-sample and isolated -PeakRSS/tracemalloc procedure as `execute_case`, but invokes the selected -function from `BASELINES`. Store `baseline_name` in the result payload and set -`backend="pandas-component-baseline"`; never route this sentinel backend -through `fd.clean`. - -- [ ] **Step 4: Implement variability and hypothesis analysis** - -`classify_change` calculates `(baseline - candidate) / baseline` and compares its absolute magnitude with both `0.10` and `2 * max(baseline_cv, candidate_cv)`. Return only `improved`, `regressed`, or `noise`. - -`classify_hypotheses` produces entries for: - -- `unnecessary_correlation` -- `repeated_null_scans` -- `repeated_uniqueness_scans` -- `copy_pressure` -- `dtype_conversion_pressure` -- `report_finalization_overhead` -- `optional_ml_overhead` -- `backend_conversion_overhead` - -Each entry contains `status` (`candidate`, `rejected`, or `insufficient_evidence`), `stage_fraction`, `observed_calls`, and an `evidence` list of exact function/file/line records. A hypothesis is only a candidate when its stage consumes at least 10% of total self time or its peak allocations consume at least 10% of the traced peak, and the relevant operation count is non-zero. - -- [ ] **Step 5: Implement deterministic Markdown rendering** - -`render_report` sorts cases by rows, width, config, report flag, backend, and output format. It renders environment data, exact commands, median/min/max/CV/throughput, peak RSS/Python/input ratios, pandas comparison ratios, stage percentages, top exact functions/lines, confirmed candidates, rejected hypotheses, failures, timeouts, OOMs, and limitations. It must not print an improvement claim for a comparison classified as `noise`. - -- [ ] **Step 6: Add `analyze` and `render` CLI commands** - -Commands: - -```bash -python -m benchmarks.performance analyze --input benchmarks/results/performance/baseline --output benchmarks/results/performance/baseline-summary.json -python -m benchmarks.performance render --input benchmarks/results/performance/baseline-summary.json --output benchmarks/results/performance/baseline-report.md -``` - -Also add: - -```bash -python -m benchmarks.performance baseline --rows 10000,100000 --widths narrow,medium,wide --dataset-types mixed --baselines shallow_copy,numeric_median_fill,duplicates,null_counts --warmups 1 --repetitions 5 --output benchmarks/results/performance/baseline -``` - -The `baseline` command accepts only names in `BASELINES` and writes the same -schema-versioned result format as `run`. - -- [ ] **Step 7: Run comparison/render tests** - -Run: `python -m pytest tests/performance/test_analysis_render.py -q --no-cov` - -Expected: PASS. - -- [ ] **Step 8: Commit analysis and rendering** - -```bash -git add benchmarks/performance/baselines.py benchmarks/performance/analysis.py benchmarks/performance/render.py benchmarks/performance/cli.py tests/performance/test_analysis_render.py -git commit -m "bench: analyze and render scalability evidence" -``` - -### Task 7: CI-Safe Contracts, Large Workflow, Make Targets, and Documentation Shell - -**Files:** -- Modify: `.gitignore` -- Modify: `Makefile` -- Create: `.github/workflows/performance-large.yml` -- Create: `docs/performance-investigation.md` -- Create: `tests/performance/test_contracts.py` -- Modify: `docs/benchmarks.md` - -**Interfaces:** -- Consumes: Tasks 1-6 CLI and schemas. -- Produces: fast PR contracts, manual/scheduled large runs, reproducible local commands, and committed compact evidence paths. - -- [ ] **Step 1: Write failing fast-contract tests** - -```python -# tests/performance/test_contracts.py -from __future__ import annotations - -import json - -from benchmarks.performance.cli import main -from benchmarks.performance.render import render_report - - -def test_ci_sized_matrix_completes_and_renders(tmp_path) -> None: - raw = tmp_path / "raw" - assert main([ - "run", "--rows", "500", "--widths", "narrow,medium", - "--configs", "default,conservative", "--report-modes", "false,true", - "--warmups", "0", "--repetitions", "1", "--timeout", "120", - "--output", str(raw), - ]) == 0 - results = [json.loads(path.read_text()) for path in raw.glob("*.json")] - assert len(results) == 8 - assert all(item["status"] == "completed" for item in results) - rendered = render_report({"environment": results[0]["environment"], "results": results, "hypotheses": {}}) - assert "500" in rendered - assert "narrow" in rendered and "medium" in rendered - - -def test_report_modes_preserve_behavioral_contract(tmp_path) -> None: - assert main([ - "run", "--rows", "250", "--widths", "narrow", "--configs", "default", - "--report-modes", "false,true", "--warmups", "0", "--repetitions", "1", - "--timeout", "120", "--output", str(tmp_path), - ]) == 0 - results = [json.loads(path.read_text()) for path in tmp_path.glob("*.json")] - assert {item["case"]["return_report"] for item in results} == {False, True} -``` - -- [ ] **Step 2: Verify the fast-contract tests pass against completed tooling** - -Run: `python -m pytest tests/performance -q --no-cov` - -Expected: PASS. If this fails, fix the responsible Task 1-6 module before changing workflow files. - -- [ ] **Step 3: Make compact evidence committable and raw results ignored** - -Replace the broad benchmark result rule with: - -```gitignore -# Benchmark runtime results: raw case files stay local; compact evidence is committed. -benchmarks/results/* -!benchmarks/results/performance/ -benchmarks/results/performance/* -!benchmarks/results/performance/*-summary.json -!benchmarks/results/performance/*-report.md -``` - -- [ ] **Step 4: Add Make targets** - -```make -.PHONY: performance-ci performance-baseline performance-profile performance-report - -performance-ci: - $(PY) -m pytest tests/performance -q --no-cov - -performance-baseline: - $(PY) -m benchmarks.performance run --output benchmarks/results/performance/baseline - -performance-profile: - $(PY) -m benchmarks.performance profile --rows 100000 --width medium --config default --report-mode true --output benchmarks/results/performance/baseline - -performance-report: - $(PY) -m benchmarks.performance analyze --input benchmarks/results/performance/baseline --output benchmarks/results/performance/baseline-summary.json - $(PY) -m benchmarks.performance render --input benchmarks/results/performance/baseline-summary.json --output benchmarks/results/performance/baseline-report.md -``` - -- [ ] **Step 5: Add the manual/scheduled large workflow** - -`.github/workflows/performance-large.yml` must run on `workflow_dispatch` and weekly schedule, install `.[dev,bench,ml]`, execute row counts `100000,500000,1000000`, widths `narrow,medium,wide`, configs `default,conservative,representation_off,statistical_off,explicit`, both report modes, one warm-up, five repetitions, then analyze/render and upload the whole performance results directory. Set `timeout-minutes: 180`; do not add it to required PR checks. - -- [ ] **Step 6: Add the documentation shell with resolved architecture content** - -Create `docs/performance-investigation.md` with these populated sections from the approved design, not empty headings: - -- Executive summary stating that baseline measurement is in progress and no slowdown/improvement is claimed yet. -- Architecture and execution flow copied from the approved design. -- Supported Python/pandas versions and default configuration. -- Reproduction commands for `performance-ci`, `performance-baseline`, `performance-profile`, and `performance-report`. -- Measurement methodology and the 10%/2x-variability rule. -- A statement that baseline, profile, root-cause, rejected-hypothesis, before/after, memory, backend, documentation, risk, and verification sections are generated from authoritative JSON as their phases complete. - -Update `docs/benchmarks.md` with links and commands for the new harness while retaining the existing CleanBench and strategic-report documentation. - -- [ ] **Step 7: Run fast performance contracts and docs build** - -Run: `python -m pytest tests/performance -q --no-cov` - -Expected: PASS. - -Run: `mkdocs build --strict` - -Expected: PASS with no missing navigation or link warnings. - -- [ ] **Step 8: Commit integration and documentation** - -```bash -git add .gitignore Makefile .github/workflows/performance-large.yml docs/performance-investigation.md docs/benchmarks.md tests/performance/test_contracts.py -git commit -m "bench: integrate large performance investigation workflow" -``` - -### Task 8: Capture the Immutable Baseline and Decide the Optimization Plans - -**Files:** -- Create: `benchmarks/results/performance/baseline-summary.json` -- Create: `benchmarks/results/performance/baseline-report.md` -- Modify: `docs/performance-investigation.md` -- Create: `docs/superpowers/specs/2026-07-11-freshdata-confirmed-performance-bottlenecks-design.md` -- Create only the applicable evidence-named plans from: - `docs/superpowers/plans/2026-07-11-freshdata-optimize-correlation.md`, - `2026-07-11-freshdata-optimize-engine-statistics.md`, - `2026-07-11-freshdata-optimize-copy-pressure.md`, - `2026-07-11-freshdata-optimize-dtype-conversions.md`, - `2026-07-11-freshdata-optimize-reporting.md`, - `2026-07-11-freshdata-optimize-optional-ml.md`, or - `2026-07-11-freshdata-optimize-backend-conversion.md`. - -**Interfaces:** -- Consumes: the clean `6f6c2fe` baseline worktree and Tasks 1-7 tooling. -- Produces: recorded commands/results, exact root causes and rejected hypotheses, and evidence-derived Phase 2 plans. - -- [ ] **Step 1: Verify the baseline identity before running** - -Run: - -```bash -git rev-parse HEAD -git status --short -python -VV -python -c "import pandas, numpy, freshdata; print(pandas.__version__, numpy.__version__, freshdata.__version__)" -``` - -Expected: the benchmarked production sources match `6f6c2fe`; documentation/tooling commits may be present, but `git diff 6f6c2fe -- src/freshdata` must be empty. The only pre-existing untracked item may be `.venv-qa/` outside the isolated execution workspace. - -- [ ] **Step 2: Run the required pandas baseline in bounded batches** - -Run these commands separately so a large-case failure does not erase smaller evidence: - -```bash -python -m benchmarks.performance run --rows 10000 --widths narrow,medium,wide --configs default,conservative,representation_off,statistical_off,explicit --report-modes false,true --warmups 1 --repetitions 5 --timeout 900 --output benchmarks/results/performance/baseline -python -m benchmarks.performance run --rows 100000 --widths narrow,medium,wide --configs default,conservative,representation_off,statistical_off,explicit --report-modes false,true --warmups 1 --repetitions 5 --timeout 1800 --output benchmarks/results/performance/baseline -python -m benchmarks.performance run --rows 500000 --widths narrow,medium,wide --configs default,conservative,representation_off,statistical_off,explicit --report-modes false,true --warmups 1 --repetitions 5 --timeout 3600 --output benchmarks/results/performance/baseline -python -m benchmarks.performance run --rows 1000000 --widths narrow,medium,wide --configs default,conservative,representation_off,statistical_off,explicit --report-modes false,true --warmups 1 --repetitions 5 --timeout 7200 --output benchmarks/results/performance/baseline -python -m benchmarks.performance run --rows 100000 --widths medium --dataset-types numeric,categorical,string,nullable,datetime,high_cardinality --configs default --report-modes false,true --warmups 1 --repetitions 5 --timeout 1800 --output benchmarks/results/performance/baseline -``` - -Expected: every case writes `completed`, `failed`, `timeout`, or `oom` JSON; no case disappears. Report all poor and incomplete outcomes. - -- [ ] **Step 3: Profile the representative bottleneck slices** - -Run: - -```bash -python -m benchmarks.performance profile --rows 10000 --width wide --config default --report-mode true --output benchmarks/results/performance/baseline -python -m benchmarks.performance profile --rows 100000 --width medium --config default --report-mode true --output benchmarks/results/performance/baseline -python -m benchmarks.performance profile --rows 500000 --width medium --config default --report-mode false --output benchmarks/results/performance/baseline -python -m benchmarks.performance profile --rows 1000000 --width narrow --config default --report-mode true --output benchmarks/results/performance/baseline -python -m benchmarks.performance profile --rows 100000 --width medium --config aggressive --report-mode true --output benchmarks/results/performance/baseline -python -m benchmarks.performance profile --rows 100000 --width medium --config semantic --report-mode true --output benchmarks/results/performance/baseline -python -m benchmarks.performance profile --rows 10000 --width medium --config missforest --report-mode true --output benchmarks/results/performance/baseline -``` - -Expected: profile JSON includes exact function/file/line records, allocation records, stage totals, and operation counts for every completed profile. - -- [ ] **Step 4: Measure comparable pandas component baselines** - -Run: - -```bash -python -m benchmarks.performance baseline --rows 10000,100000,500000,1000000 --widths narrow,medium,wide --dataset-types mixed --baselines shallow_copy,numeric_median_fill,duplicates,null_counts --warmups 1 --repetitions 5 --timeout 3600 --output benchmarks/results/performance/baseline -``` - -Expected: component comparisons are labelled by exact operation; no result claims to replicate the full balanced decision/audit pipeline. - -- [ ] **Step 5: Analyze and render compact evidence** - -Run: - -```bash -python -m benchmarks.performance analyze --input benchmarks/results/performance/baseline --output benchmarks/results/performance/baseline-summary.json -python -m benchmarks.performance render --input benchmarks/results/performance/baseline-summary.json --output benchmarks/results/performance/baseline-report.md -``` - -Expected: the summary includes environment, commands, all statuses, median/min/max/CV, peak RSS/Python memory, ratios, stage shares, exact hot functions/lines, and eight hypothesis decisions. - -- [ ] **Step 6: Update the investigation report from authoritative evidence** - -Copy generated tables and findings without hand-editing numeric values. Populate these sections in `docs/performance-investigation.md`: - -1. Executive summary limited to baseline findings. -2. Reproduction commands. -3. Baseline benchmark table. -4. Profiling findings with exact functions/files/lines. -5. Confirmed root causes. -6. Rejected hypotheses. -12. Baseline peak-memory table. -13. Remaining baseline bottlenecks. -16. Baseline risks and limitations. - -Leave before/after optimization sections explicitly marked `not yet applicable: no production optimization has been implemented`, which is a resolved status rather than an unknown placeholder. - -- [ ] **Step 7: Write the evidence-derived Phase 2 design and plans** - -Create `docs/superpowers/specs/2026-07-11-freshdata-confirmed-performance-bottlenecks-design.md` containing only hypotheses classified `candidate` by `baseline-summary.json`. For each candidate, include its exact functions/lines, affected cases, measured time/memory fraction, behavioral invariants, proposed smallest change, alternative rejected approaches, and benchmark acceptance case. - -Then run the brainstorming approval gate for that design. After approval, create one detailed TDD plan per independently rejectable root cause. Do not create or execute a production optimization plan for any hypothesis classified `rejected` or `insufficient_evidence`. - -- [ ] **Step 8: Verify and commit baseline evidence** - -Run: - -```bash -python -m pytest tests/performance tests/benchmark -q --no-cov -ruff check benchmarks/performance tests/performance -mypy benchmarks/performance -mkdocs build --strict -git diff --check -``` - -Expected: all commands pass. Raw case files remain ignored; compact summary/report, investigation document, and evidence-derived design/plans are staged. - -```bash -git add benchmarks/results/performance/baseline-summary.json benchmarks/results/performance/baseline-report.md docs/performance-investigation.md docs/superpowers/specs/2026-07-11-freshdata-confirmed-performance-bottlenecks-design.md docs/superpowers/plans -git commit -m "bench: record FreshData scalability baseline" -``` - -## Phase 1 Completion Gate - -Phase 1 is complete only when: - -- The required 10k, 100k, 500k, and 1M row matrix has a recorded status for every requested width/config/report case. -- Median, minimum, maximum, variability, peak RSS, Python allocations, input ratio, and comparable pandas ratios are recorded. -- Exact functions, files, and lines are recorded for the largest time and allocation costs. -- Context/cache, correlation, missing, outlier, role, dtype, duplicate, audit, report, ML/semantic, conversion, and backend costs are separated where exercised. -- Copy/conversion/scan counts are explicitly labelled observed calls. -- Confirmed root causes and rejected hypotheses are evidence-backed. -- No production cleaning code differs from `6f6c2fe`. -- The Phase 2 design includes only confirmed causes and has been presented for user approval. - -Do not claim the overall FreshData performance goal complete at this gate. The active goal remains open through production optimization, before/after benchmarks, backend recommendations, documentation correction, full compatibility verification, and the final 17-deliverable audit. diff --git a/docs/superpowers/plans/2026-07-13-freshdata-semantic-performance-optimization.md b/docs/superpowers/plans/2026-07-13-freshdata-semantic-performance-optimization.md deleted file mode 100644 index 5865511..0000000 --- a/docs/superpowers/plans/2026-07-13-freshdata-semantic-performance-optimization.md +++ /dev/null @@ -1,440 +0,0 @@ -# FreshData Semantic Performance Optimization Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox syntax for tracking. - -**Goal:** Either ship a verified request-local semantic-context reuse optimization, or publish a measured rejection without expanding scope. - -**Architecture:** A benchmark-only probe measures only the seven approved context operations and resets for every context build. If within-build repetition is material, build_semantic_context owns one private memo shared only with its _build_info calls. Separately, analyzer field validators remove the six known mypy errors without changing the JSON contract. - -**Tech Stack:** Python 3.9+, pandas 1.5–<3, pytest, unittest.mock, cProfile/tracemalloc harness, Ruff, mypy, MkDocs. - -## Global Constraints - -- Preserve Python >=3.9, pandas >=1.5,<3, NumPy >=1.21, public APIs/configuration, and add no dependency. -- Preserve dataframe values/order/index/dtypes, input mutation, warnings/errors, report/audit serialization and ordering, policies/protected columns, semantic modes, memory/profile replay, and native/fallback behavior. -- Scope is only case 9ea9cb03dfc114c5 and its semantic-context candidate. Do not optimize MissForest, correlation, backend conversion, copy, dtype, reporting, uniqueness, or null scanning. -- The operation universe is exactly is_plain_number, parse_number_words, post-str(v) parse_boolean, parse_currency, parse_unit, email-shape matching, and looks_like_date_value. -- Cache exact built-in str/int/float/bool only for is_plain_number and exact built-in str only for every other operation. Bypass every other value before hashing/equality. -- Cache lifetime is one build_semantic_context call. No global/lru cache, public switch, cross-request state, dependency, or speculative size cap. -- Cache only successful immutable results; never cache/suppress/reorder exceptions. -- Accept only a >=10% median primary-case win exceeding 2x the larger CV, exact output/report fingerprints, no material memory regression, no meaningful default/aggressive control regression, and a confirming profile. - ---- - -## File Structure - -| Path | Responsibility | -| --- | --- | -| benchmarks/performance/semantic_probe.py | Development-only single-build instrumentation; never imported by production FreshData. | -| tests/performance/test_semantic_probe.py | Probe reset, keying, conversion, and bypass tests. | -| src/freshdata/semantic/context.py | Conditional private one-build memo and unchanged context scoring. | -| tests/test_semantic_cleaning.py | Memo lifetime, key separation, bypass, exception, and public behavior tests. | -| benchmarks/performance/analysis.py | Runtime-checked typed analyzer helpers. | -| tests/performance/test_analysis_render.py | Analyzer malformed-record regressions. | -| docs/performance-investigation.md | Actual Phase 2 evidence and accepted/rejected result. | - -### Task 1: Add the development-only semantic repetition probe - -**Files:** - -- Create: benchmarks/performance/semantic_probe.py -- Create: tests/performance/test_semantic_probe.py - -**Interfaces:** - -- Consumes: freshdata.semantic.context.build_semantic_context, the seven context operation references, pandas.DataFrame, CleanConfig. -- Produces: SemanticProbeBuild containing per-operation total_calls, eligible_calls, bypassed_calls, unique_keys, theoretical_hits, hit_rate, and in-memory eligible value sequence; probe_context_build(df, config, *, stats=None). -- Does not modify production code, the benchmark schema, or the CLI. - -- [ ] **Step 1: Write failing probe tests** - -~~~python -def test_probe_counts_only_within_one_context_build() -> None: - frame = pd.DataFrame({ - "left": ["yes", "no", "2024-01-01"], - "right": ["yes", "no", "2024-01-01"], - }) - config = CleanConfig(semantic_mode="assist", verbose=False) - - _context, first = probe_context_build(frame, config) - _context, second = probe_context_build(frame, config) - - assert first.by_operation["parse_boolean"].theoretical_hits >= 1 - assert second.total_theoretical_hits == first.total_theoretical_hits - - -def test_probe_uses_exact_types_and_bypasses_unsafe_values() -> None: - frame = pd.DataFrame({ - "ints": pd.Series([1], dtype=object), - "bools": pd.Series([True], dtype=object), - "text": pd.Series(["1"], dtype=object), - }) - _context, result = probe_context_build( - frame, CleanConfig(semantic_mode="assist", verbose=False) - ) - - numeric = result.by_operation["is_plain_number"] - assert {type(value) for value in numeric.eligible_values} == {int, bool, str} - assert numeric.unique_keys == 3 -~~~ - -Also add direct-helper tests proving a list is bypassed without hashing and that parse_boolean records the post-str(v) string. - -- [ ] **Step 2: Verify RED** - -Run: .venv-qa/bin/python -m pytest tests/performance/test_semantic_probe.py -q --no-cov - -Expected: FAIL because the module does not exist. - -- [ ] **Step 3: Implement the minimal probe** - -~~~python -@dataclass(frozen=True) -class OperationProbe: - total_calls: int - eligible_calls: int - bypassed_calls: int - unique_keys: int - theoretical_hits: int - eligible_values: tuple[object, ...] - - @property - def hit_rate(self) -> float: - return self.theoretical_hits / self.eligible_calls if self.eligible_calls else 0.0 - - -def _eligible(operation: str, value: object) -> bool: - return type(value) in _ALLOWED_TYPES[operation] - - -def probe_context_build( - df: pd.DataFrame, - config: CleanConfig, - *, - stats: dict[object, tuple[int, int, int | None]] | None = None, -) -> tuple[SemanticContext, SemanticProbeBuild]: - with _patched_context_operations() as probe: - context = semantic_context.build_semantic_context(df, config, stats=stats) - return context, probe.finish_build() -~~~ - -Patch the six imported callable references in freshdata.semantic.context plus an _EMAIL_VALUE proxy whose match method records the already-stripped value then delegates. Eligible keys are (operation, type(value), value). A new probe is created for every function call, all raw keys remain in memory only, and bypassed values are never hashed. - -- [ ] **Step 4: Verify GREEN** - -Run: .venv-qa/bin/python -m pytest tests/performance/test_semantic_probe.py -q --no-cov && .venv-qa/bin/ruff check benchmarks/performance/semantic_probe.py tests/performance/test_semantic_probe.py - -Expected: PASS. - -- [ ] **Step 5: Commit** - -~~~bash -git add benchmarks/performance/semantic_probe.py tests/performance/test_semantic_probe.py -git commit -m "test: add semantic context repetition probe" -~~~ - -### Task 2: Record the hard discovery decision - -**Files:** - -- Modify: docs/performance-investigation.md -- Create locally only: .superpowers/sdd/task-9-semantic-discovery.md - -**Interfaces:** - -- Consumes: Task 1 and make_mixed_frame(DatasetSpec(rows=100000, width="medium", seed=42, dataset_type="mixed")) with CleanConfig(semantic_mode="assist", verbose=False). -- Produces: a committed entry with the command, environment, per-operation metrics, and accepted_for_implementation or rejected_no_material_within_build_reuse. -- Decision boundary: only hits within one build count; warm-up, measurement, memory, profiling, and all cross-build repetitions never count. - -- [ ] **Step 1: Add a focused discovery fixture test** - -~~~python -def test_probe_reports_reuse_for_repeated_context_values() -> None: - frame = pd.DataFrame({ - "one": ["yes", "no", "2024-01-01"], - "two": ["yes", "no", "2024-01-01"], - }) - _context, result = probe_context_build( - frame, CleanConfig(semantic_mode="assist", verbose=False) - ) - assert result.total_theoretical_hits > 0 - assert result.by_operation["looks_like_date_value"].theoretical_hits > 0 -~~~ - -- [ ] **Step 2: Run the test and single-build primary measurement** - -Run: .venv-qa/bin/python -m pytest tests/performance/test_semantic_probe.py::test_probe_reports_reuse_for_repeated_context_values -q --no-cov - -Then, serially: - -~~~bash -PYTHONPATH=src .venv-qa/bin/python -c 'from benchmarks.performance.datasets import DatasetSpec, make_mixed_frame; from benchmarks.performance.semantic_probe import probe_context_build; from freshdata.config import CleanConfig; frame = make_mixed_frame(DatasetSpec(rows=100000, width="medium", seed=42, dataset_type="mixed")); _, result = probe_context_build(frame, CleanConfig(semantic_mode="assist", verbose=False)); print(result.to_json())' -~~~ - -Expected: valid JSON for one context build. - -- [ ] **Step 3: Make the explicit branch decision** - -Advance only if repeated eligible calls occur in measured hot operations and their timed repeat work makes a 10% end-to-end win plausible. Otherwise skip Task 3, record rejection, do not change src/freshdata/semantic/context.py, and continue to Tasks 4 and 5. - -- [ ] **Step 4: Document and commit actual evidence** - -Add the exact command, commit/dirty state, each per-operation metric, decision, and reason to docs/performance-investigation.md. A rejected result must say no production optimization was implemented; an accepted result must say it only passed discovery. - -Run: .venv-qa/bin/mkdocs build --strict && git diff --check - -~~~bash -git add docs/performance-investigation.md -git commit -m "docs: record semantic discovery decision" -~~~ - -### Task 3: Conditionally add request-local reuse - -**Files:** - -- Modify: src/freshdata/semantic/context.py:43-253 -- Modify: tests/test_semantic_cleaning.py:349-550 - -**Interfaces:** - -- Consumes: Task 2's explicitly accepted operation subset. -- Produces: private _SemanticContextMemo.call(operation, value, fn), constructed once in build_semantic_context and passed to _build_info. -- Guarantees: operation/type/value key separation; pre-hash bypass; successful immutable results only; no cross-build state or cached exceptions. - -- [ ] **Step 1: Write failing cache and equivalence tests** - -~~~python -def test_context_memo_reuses_only_within_one_memo() -> None: - memo = semantic_context._SemanticContextMemo() - calls = 0 - - def observed(value: object) -> bool: - nonlocal calls - calls += 1 - return semantic_context.is_plain_number(value) - - assert memo.call("is_plain_number", "12", observed) is True - assert memo.call("is_plain_number", "12", observed) is True - assert calls == 1 - assert semantic_context._SemanticContextMemo().call( - "is_plain_number", "12", observed - ) is True - assert calls == 2 - - -def test_context_memo_distinguishes_bool_int_and_str() -> None: - memo = semantic_context._SemanticContextMemo() - calls: list[object] = [] - - def observed(value: object) -> bool: - calls.append(value) - return semantic_context.is_plain_number(value) - - assert memo.call("is_plain_number", 1, observed) is True - assert memo.call("is_plain_number", True, observed) is False - assert memo.call("is_plain_number", "1", observed) is True - assert calls == [1, True, "1"] -~~~ - -Add failing tests for list bypass with two underlying calls, two identical raised -exceptions, and a repeated semantic fixture for the exact operation selected by -Task 2. That fixture must prove the selected call site invokes its underlying -operation once per repeated key within one build and again in a second build, -then assert the exact cleaned dataframe/report action fingerprint. Parameterize -public cleaning assertions over off, assist, review, and auto and retain -protected-column expectations. - -- [ ] **Step 2: Verify RED** - -Run: .venv-qa/bin/python -m pytest tests/test_semantic_cleaning.py -q --no-cov -k "context_memo or repeated_values or protected or semantic" - -Expected: FAIL because _SemanticContextMemo does not exist. - -- [ ] **Step 3: Implement the minimal private memo** - -~~~python -_MISSING = object() - -class _SemanticContextMemo: - def __init__(self) -> None: - self._values: dict[tuple[str, type[object], object], object] = {} - - def call(self, operation: str, value: object, fn: Callable[[Any], T]) -> T: - if type(value) not in _ALLOWED_TYPES[operation]: - return fn(value) - key = (operation, type(value), value) - cached = self._values.get(key, _MISSING) - if cached is not _MISSING: - return cast(T, cached) - result = fn(value) - self._values[key] = result - return result -~~~ - -Introduce _email_shape(value: str) returning exactly bool(_EMAIL_VALUE.match(value.strip())). Preserve post-str(v) Boolean conversion by memo.call("parse_boolean", str(v), parse_boolean). Instantiate the memo only inside build_semantic_context, pass it to _build_info, and enable only Task 2's proven operation subset. - -- [ ] **Step 4: Verify behavior** - -Run: .venv-qa/bin/python -m pytest tests/test_semantic_cleaning.py tests/test_semantic_backends.py tests/test_execution/test_native_semantic.py tests/learning/test_replay.py -q --no-cov - -Expected: PASS. Any changed report ordering, exception, policy result, or native parity is a stop-and-revert condition. - -- [ ] **Step 5: Commit** - -~~~bash -git add src/freshdata/semantic/context.py tests/test_semantic_cleaning.py -git commit -m "perf: reuse semantic context predicate results" -~~~ - -### Task 4: Fix the six inherited analyzer mypy errors - -**Files:** - -- Modify: benchmarks/performance/analysis.py:12-252 -- Modify: tests/performance/test_analysis_render.py - -**Interfaces:** - -- Consumes: schema-validated dict[str, object] profiles. -- Produces: HypothesisRule and checked numeric/record helpers while retaining output ordering, thresholds, and JSON semantics. - -- [ ] **Step 1: Write failing malformed-profile tests** - -~~~python -def test_hypothesis_classifier_rejects_non_integer_profile_line() -> None: - profile = _empty_profile() - profile["functions"] = [{ - "file": "src/freshdata/engine/context.py", - "line": "bad", - "function": "build_context", - "self_seconds": 0.0, - "cumulative_seconds": 0.0, - "calls": 1, - }] - with pytest.raises(TypeError, match="profile function line must be an integer"): - classify_hypotheses(profile) -~~~ - -Add matching assertions for non-finite stage values and non-integer operation/call counts. Existing valid-profile classifier tests remain unchanged. - -- [ ] **Step 2: Verify RED** - -Run: .venv-qa/bin/python -m pytest tests/performance/test_analysis_render.py -q --no-cov - -Expected: FAIL because malformed fields currently fall through to int/float conversion. - -- [ ] **Step 3: Add checked types and helpers** - -~~~python -class HypothesisRule(TypedDict): - stages: tuple[str, ...] - operations: tuple[str, ...] - evidence: tuple[tuple[str, tuple[str, ...]], ...] - - -def _integer_field(value: object, label: str) -> int: - if isinstance(value, bool) or not isinstance(value, int): - raise TypeError(f"{label} must be an integer") - return value - - -def _finite_number_field(value: object, label: str) -> float: - if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value): - raise TypeError(f"{label} must be a finite number") - return float(value) -~~~ - -Annotate _HYPOTHESES as dict[str, HypothesisRule]. Check and narrow function/allocation arrays before using them. Use the helpers for lines, stage values, operations, calls, and allocation bytes; do not use a cast to evade validation. - -- [ ] **Step 4: Verify GREEN** - -Run: .venv-qa/bin/python -m pytest tests/performance/test_analysis_render.py -q --no-cov && .venv-qa/bin/mypy benchmarks/performance/analysis.py && .venv-qa/bin/ruff check benchmarks/performance/analysis.py tests/performance/test_analysis_render.py - -Expected: PASS and mypy reports Success: no issues found. - -- [ ] **Step 5: Commit** - -~~~bash -git add benchmarks/performance/analysis.py tests/performance/test_analysis_render.py -git commit -m "fix: validate performance analysis record fields" -~~~ - -### Task 5: Run acceptance, publish the Phase 2 outcome, and verify completion - -**Files:** - -- Modify: docs/performance-investigation.md -- Modify only if regenerated: benchmarks/results/performance/baseline-summary.json -- Modify only if regenerated: benchmarks/results/performance/baseline-report.md - -**Interfaces:** - -- Consumes: Task 2 decision, Task 3 if accepted, primary/control benchmark JSON, and a confirming profile. -- Produces: evidence-backed accepted or rejected Phase 2 result, never an unsupported performance claim. - -- [ ] **Step 1: Capture clean serial benchmark evidence** - -Before Task 3, run: - -~~~bash -.venv-qa/bin/python -m benchmarks.performance run --rows 100000 --widths medium --configs semantic --report-modes true --backends pandas --output-formats pandas --seed 42 --warmups 1 --repetitions 5 --output /private/tmp/freshdata-semantic-before -~~~ - -After Task 3, run serially: - -~~~bash -.venv-qa/bin/python -m benchmarks.performance run --rows 100000 --widths medium --configs semantic,default,aggressive --report-modes true --backends pandas --output-formats pandas --seed 42 --warmups 1 --repetitions 5 --output /private/tmp/freshdata-semantic-after -~~~ - -If Task 2 rejects, skip before/after runs and retain the discovery evidence. - -- [ ] **Step 2: Apply the acceptance gate** - -For an accepted implementation, require equal output_fingerprint, report_fingerprint, and result_type; classify timing with classify_change(before_median, after_median, before_cv, after_cv); inspect raw samples, RSS, Python allocation peak, and both controls. Revert Task 3 if any required gate fails. - -- [ ] **Step 3: Capture a confirming profile** - -~~~bash -.venv-qa/bin/python -m benchmarks.performance profile --rows 100000 --widths medium --configs semantic --report-modes true --backends pandas --output-formats pandas --seed 42 --warmups 1 --repetitions 5 --output /private/tmp/freshdata-semantic-profile-after -~~~ - -Confirm targeted context-operation work decreases without merely shifting to another stage. - -- [ ] **Step 4: Publish actual, not projected, evidence** - -Update docs/performance-investigation.md with command, environment, dirty state, metrics, fingerprints, profile delta, and accepted/rejected conclusion. If rejected, retain the no-production-optimization statement and explain why. Regenerate compact result/report artifacts only when their schema stays valid and no duplicate case rows are introduced. - -- [ ] **Step 5: Final verification and commit** - -Run serially: - -~~~bash -.venv-qa/bin/python -m pytest tests/test_semantic_cleaning.py tests/test_semantic_backends.py tests/test_execution/test_native_semantic.py tests/learning/test_replay.py tests/performance -q --no-cov -.venv-qa/bin/ruff check src tests benchmarks/performance -.venv-qa/bin/mypy src/freshdata -.venv-qa/bin/mypy benchmarks/performance/analysis.py -.venv-qa/bin/mkdocs build --strict -git diff --check -.venv-qa/bin/python -m pytest -~~~ - -Commit intended documentation/compact artifacts only: - -~~~bash -git add docs/performance-investigation.md benchmarks/results/performance/baseline-summary.json benchmarks/results/performance/baseline-report.md -git commit -m "docs: publish semantic optimization outcome" -~~~ - -Omit unchanged paths. Never commit raw /private/tmp results, .venv-qa, or .superpowers/sdd reports. - -## Review Gates - -After each task, create an exact base..HEAD review package, use a fresh read-only reviewer, fix every Critical or Important finding, then re-review the correction range. The final review checks the accepted/rejected evidence chain, production diff, static checks, links, and intended worktree. - -## Plan Self-Review - -- Tasks 1–2 implement the mandatory per-build discovery gate. -- Task 3 is the sole conditional production optimization. -- Task 4 resolves exactly the six inherited analysis.py mypy errors, not unrelated benchmark typing debt. -- Task 5 enforces equivalence, timing, memory, controls, profile, documentation, and complete verification. -- Every code task includes paths, interfaces, RED, GREEN, commands, and a commit. diff --git a/docs/superpowers/plans/2026-07-14-freshdata-truthbench.md b/docs/superpowers/plans/2026-07-14-freshdata-truthbench.md deleted file mode 100644 index 5a77975..0000000 --- a/docs/superpowers/plans/2026-07-14-freshdata-truthbench.md +++ /dev/null @@ -1,687 +0,0 @@ -# FreshData TruthBench Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Build a deterministic, privacy-safe semantic red-team and regression system that gives every test cell a gold disposition, exercises every public FreshData decision surface, minimizes failures, and blocks releases on the eight approved safety gates. - -**Architecture:** Add an independent `benchmarks.truthbench` package with immutable oracle models, eight deterministic domain fixtures, exact typed comparison, public-surface adapters, normalized records, privacy scanning, absolute gates, repeat/backend comparison, generated-code isolation, deterministic minimization, and atomic result reporting. Keep all model-assisted activity outside FreshData's runtime; release runs call only deterministic public APIs and Copilot with `provider=None`. Source fixes are allowed only after a TruthBench reproduction and a focused failing pytest regression establish the defect. - -**Tech Stack:** Python 3.9+, pandas, Polars, DuckDB, FreshData public APIs, dataclasses/enums, hashlib/HMAC, `jsonschema`, `ast`, `subprocess`, pytest, GitHub Actions. - -## Global Constraints - -- Work only in `/Users/wilson/freshdata-worktrees/truthbench-jwd` on `feature/truthbench-jwd`; do not modify `/Users/wilson/freshdata`. -- Keep GPT/model/provider calls out of `fd.clean`, semantic routing, domain repair, privacy processing, trust scoring, CI, and TruthBench release execution. -- Do not weaken, skip, rebaseline, or rewrite existing tests or gold outputs to obtain a pass. -- Retain the approved baseline note: the two one-off `tests/test_benchmark.py` throughput failures are pre-existing environment timing noise; both exact reruns passed. Do not change their thresholds. -- A required backend that is unavailable or falls back is a failure, not a skip. -- Never write raw fixture PII into result JSON, Markdown, logs, exceptions, minimized artifacts, or committed baselines. -- Use one focused commit per task or validated source defect. Run the named focused test before every commit. -- For every implementation defect: reproduce, minimize, explain root cause, add a permanent regression, fix the root cause, rerun the affected suite, and update TruthBench artifacts. - -## Planned File Structure - -```text -benchmarks/truthbench/ - __init__.py # supported public harness API - __main__.py # module entry point - cli.py # release/extended/check command line - models.py # immutable oracle/result/gate models - exact.py # typed exact values and equality - schema.py # JSON schema and aggregate validation - privacy.py # canary variants, redaction, sink scanner - inventory.py # classified public-surface manifest - fixtures/ - __init__.py # registry and build_fixture - base.py # fixture builder and completeness validation - finance.py - healthcare.py - retail.py - crm.py - logistics.py - government.py - education.py - insurance.py - surfaces/ - __init__.py # adapter registry - base.py # adapter protocol and observation envelope - cleaning.py # clean/Cleaner/CSV/pipeline/plan/streaming - validation.py # field/suite/context/domain/contracts/text - privacy.py # PII/anonymization/privacy policy - reporting.py # reports/findings/export/render/CLI sinks - backends.py # pandas/Polars/DuckDB parity execution - copilot.py # provider=None and generated-code harness - normalize.py # observations -> per-cell decision records - gates.py # absolute release gates - determinism.py # stable decision hashes and repeat comparison - generated_code.py # offline AST/compile/subprocess execution - minimize.py # deterministic one-minimal failure reducer - runner.py # end-to-end orchestration - report.py # atomic JSON/Markdown/baseline output - results/ - README.md - baseline.json - latest.json - latest.md - failures/.gitkeep -tests/truthbench/ - conftest.py - test_models_exact.py - test_fixtures.py - test_privacy.py - test_inventory.py - test_surface_adapters.py - test_normalize.py - test_gates.py - test_backends_determinism.py - test_generated_code.py - test_minimize.py - test_runner_cli.py -tests/regressions/ - test_truthbench_domain_guard.py - test_truthbench_domain_report.py - test_truthbench_quarantine.py - test_truthbench_semantic_review.py - test_truthbench_findings_audit.py -docs/truthbench.md -.github/workflows/truthbench-extended.yml -``` - -Existing files changed only where the failing tests justify it: `src/freshdata/api.py`, `src/freshdata/report.py`, `src/freshdata/steps/dtypes.py`, `src/freshdata/engine/missing.py`, `src/freshdata/semantic/policy.py`, `src/freshdata/findings.py`, `Makefile`, `.github/workflows/ci.yml`, and `.github/workflows/release.yml`. - ---- - -### Task 1: Establish the immutable oracle and exact typed comparator - -**Files:** -- Create: `benchmarks/truthbench/__init__.py` -- Create: `benchmarks/truthbench/models.py` -- Create: `benchmarks/truthbench/exact.py` -- Create: `tests/truthbench/test_models_exact.py` - -**Interfaces:** Consumes Python/pandas scalar values and fixture metadata. Produces frozen `GoldCell`, `CaseExpectation`, `DecisionRecord`, `GateResult`, and `RunResult` objects plus canonical JSON-safe typed values. No FreshData runtime dependency. - -- [ ] Write failing tests covering the four-only disposition enum, stable cell IDs, sensitive-value redaction, JSON round trips, and exact distinctions among `"402.10"`, `402.1`, `np.float64(402.1)`, `None`, `pd.NA`, `NaN`, NFC/NFD Unicode, timezone-aware timestamps, leading-zero IDs, and categorical/string dtypes. - -```python -def test_exact_values_do_not_use_gauntlet_canonicalization(): - assert not exact_equal("402.10", 402.1) - assert not exact_equal(" AAPL", "AAPL") - assert not exact_equal("AAPL", "aapl") - assert exact_equal(pd.NA, pd.NA) - -def test_sensitive_record_never_serializes_raw_value(): - cell = GoldCell.create("v1", "crm", "r7", "notes", "flag", sensitive=True) - record = DecisionRecord.for_test(cell=cell, input_value="tb.person+7@example.invalid") - payload = record.to_dict() - assert "tb.person+7@example.invalid" not in json.dumps(payload) - assert payload["input"]["display"] == "[REDACTED]" -``` - -- [ ] Run `PYTHONPATH=src python -m pytest tests/truthbench/test_models_exact.py -q`; expect import/collection failure because the package does not exist. - -- [ ] Implement `Disposition(StrEnum)`, frozen dataclasses with explicit `schema_version=1`, `GoldCell.create()` stable ID generation, `TypedValue`, and a `canonical_json()` serializer that rejects non-finite JSON numbers and unknown types. - -```python -class Disposition(str, Enum): - PRESERVE = "preserve" - REPAIR = "repair" - FLAG = "flag" - REVIEW = "review" - -def exact_equal(left: Any, right: Any) -> bool: - return encode_typed(left) == encode_typed(right) - -def stable_digest(value: Any, *, key: bytes) -> str: - encoded = canonical_json(encode_typed(value)).encode("utf-8") - return hmac.new(key, encoded, hashlib.sha256).hexdigest() -``` - -- [ ] Ensure every `DecisionRecord` includes expected/actual disposition, input/output type, confidence, rationale, audit, trust, requested/actual backend, fallback events, and repeat hash fields; represent non-applicable dimensions explicitly as `None`, never by omission. - -- [ ] Rerun the focused test; expect all tests to pass. - -- [ ] Commit: `git add benchmarks/truthbench tests/truthbench/test_models_exact.py && git commit -m "feat: add TruthBench oracle models"` - -### Task 2: Add JSON schema and aggregate integrity validation - -**Files:** -- Create: `benchmarks/truthbench/schema.py` -- Create: `tests/truthbench/test_schema.py` - -**Interfaces:** Consumes serialized `RunResult`. Produces either a validated payload or a precise `TruthBenchSchemaError`. It must reject partial runs and inconsistent aggregates before gates are evaluated. - -- [ ] Write failing tests for unknown schema versions, absent fixtures/backends/gates, duplicate record IDs, non-finite confidence/trust values, fixture-hash mismatches, incorrect aggregate counts, and an `overall_passed=True` claim with a failed gate. - -- [ ] Run `PYTHONPATH=src python -m pytest tests/truthbench/test_schema.py -q`; expect failures for missing schema validation. - -- [ ] Implement a Draft 2020-12 schema with `additionalProperties: false` at result, record, gate, failure, and environment levels. Add semantic post-validation: - -```python -def validate_run(payload: Mapping[str, Any]) -> None: - jsonschema.Draft202012Validator(RESULT_SCHEMA).validate(payload) - ids = [record["record_id"] for record in payload["records"]] - if len(ids) != len(set(ids)): - raise TruthBenchSchemaError("duplicate decision record id") - if payload["summary"]["records"] != len(ids): - raise TruthBenchSchemaError("record aggregate does not match records") - passed = all(gate["passed"] for gate in payload["gates"]) - if payload["summary"]["overall_passed"] is not passed: - raise TruthBenchSchemaError("overall gate claim is inconsistent") -``` - -- [ ] Rerun the focused test; expect pass. - -- [ ] Commit: `git add benchmarks/truthbench/schema.py tests/truthbench/test_schema.py && git commit -m "feat: validate TruthBench result integrity"` - -### Task 3: Build fixture infrastructure with complete physical-cell labels - -**Files:** -- Create: `benchmarks/truthbench/fixtures/__init__.py` -- Create: `benchmarks/truthbench/fixtures/base.py` -- Create: `tests/truthbench/conftest.py` -- Create: `tests/truthbench/test_fixtures.py` - -**Interfaces:** Consumes a domain name and fixed seed. Produces a `TruthFixture` holding pristine/adversarial pandas frames, one `GoldCell` for every physical adversarial-frame cell, schema, policy, protected columns, PII canaries, row cases, schema cases, and a deterministic fixture hash. - -- [ ] Write failing fixture invariant tests. The label count must equal `rows * columns`; every `(row_id, column)` appears exactly once; injected cases overwrite the default `preserve` label; every repair has an exact typed output; every sensitive cell has a canary ID; row/schema expectations never stand in for cell labels. - -```python -@pytest.mark.parametrize("domain", DOMAINS) -def test_every_physical_cell_has_exactly_one_label(domain): - fixture = build_fixture(domain, seed=1729) - expected = {(str(row), str(col)) for row in fixture.frame.index for col in fixture.frame} - actual = {(cell.row_id, cell.column) for cell in fixture.cells} - assert actual == expected - assert len(fixture.cells) == len(expected) - fixture.validate() -``` - -- [ ] Run `PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q`; expect import failure. - -- [ ] Implement `FixtureBuilder` so it starts with a complete `preserve` label matrix and `inject()` atomically changes a value and replaces exactly one label. Reject missing/duplicate row IDs, unknown columns, non-synthetic PII domains, and contradictory repair outputs. - -```python -class FixtureBuilder: - def inject(self, row_id: str, column: str, value: Any, disposition: Disposition, - *, expected: Any = UNSET, family: str, sensitive: bool = False) -> None: - key = (row_id, column) - if key not in self._labels: - raise FixtureError(f"unknown cell {key}") - self.frame.at[row_id, column] = value - self._labels[key] = GoldCell.create( - self.version, self.domain, row_id, column, disposition, - expected_output=expected, family=family, sensitive=sensitive, - ) -``` - -- [ ] Make `fixture_hash` cover typed pristine/adversarial values, labels, schema, policy, row/schema cases, and protected columns, excluding object identity and build time. - -- [ ] Rerun the focused test with a temporary minimal fixture registered in `conftest.py`; expect pass. - -- [ ] Commit: `git add benchmarks/truthbench/fixtures tests/truthbench/conftest.py tests/truthbench/test_fixtures.py && git commit -m "feat: add complete TruthBench fixture oracle"` - -### Task 4: Add finance, healthcare, retail, and CRM gold datasets - -**Files:** -- Create: `benchmarks/truthbench/fixtures/finance.py` -- Create: `benchmarks/truthbench/fixtures/healthcare.py` -- Create: `benchmarks/truthbench/fixtures/retail.py` -- Create: `benchmarks/truthbench/fixtures/crm.py` -- Modify: `benchmarks/truthbench/fixtures/__init__.py` -- Modify: `tests/truthbench/test_fixtures.py` - -**Interfaces:** Each `build(seed: int) -> TruthFixture` is pure and deterministic. Cases use only reserved synthetic namespaces such as `.invalid`, `555-01xx`, and explicit `TB-*` identifiers. - -- [ ] Add failing domain-content tests requiring each disposition in every domain and the following exact adversarial families: - -| Domain | Required cases | -|---|---| -| finance | `apple` in price, `Apple` company, `AAPL` ticker; `0.00`; negative/extreme values; `₹1,23,456.70`; USD/EUR/INR conflict; `01/02/2025`; invisible PII in memo; tail-row account canary; protected ticker policy conflict | -| healthcare | valid rare ICD/LOINC; `98.6` in Celsius; `5 mg`/`5000 mcg`; partial/FHIR/impossible dates; decomposed Unicode name; MRN/phone/PHI in notes; protected DOB repair conflict | -| retail | leading-zero SKU/GTIN; free item and return quantity; mixed decimal/grouping/currency; HTML/entity/mojibake review; multilingual product; card/email in review; added/reordered/type-drifted columns | -| CRM | Unicode/combining names; `.invalid` email; spaced phone; ambiguous country/language/date; lifecycle contradiction; zero-width email and hidden SSN; `apple` lead source versus `Apple` employer; protected customer ID | - -- [ ] Run `PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q`; expect four missing builders/content failures. - -- [ ] Implement 16-row pristine frames per domain with stable string indexes and at least 12 injected cases per domain. Use fixed reference date `2026-01-15`, UTC timezone, and explicit locale metadata. Keep unusual valid values labelled `preserve`, deterministic representational fixes labelled `repair`, unsafe values labelled `flag`, and ambiguous/policy-conflicted values labelled `review`. - -- [ ] Add row cases for exact duplicates and schema cases for added/removed/renamed/reordered/type-drifted columns. Do not label a removed row/column as a cell outcome. - -- [ ] Assert byte-for-byte deterministic frame serialization and fixture hashes over two builds for seeds `1729` and `2718`. - -- [ ] Rerun the fixture tests; expect pass. - -- [ ] Commit: `git add benchmarks/truthbench/fixtures tests/truthbench/test_fixtures.py && git commit -m "feat: add first TruthBench domain corpus"` - -### Task 5: Add logistics, government, education, and insurance gold datasets - -**Files:** -- Create: `benchmarks/truthbench/fixtures/logistics.py` -- Create: `benchmarks/truthbench/fixtures/government.py` -- Create: `benchmarks/truthbench/fixtures/education.py` -- Create: `benchmarks/truthbench/fixtures/insurance.py` -- Modify: `benchmarks/truthbench/fixtures/__init__.py` -- Modify: `tests/truthbench/test_fixtures.py` - -**Interfaces:** Same builder contract as Task 4; registry order is the stable alphabetical order `crm, education, finance, government, healthcare, insurance, logistics, retail`. - -- [ ] Add failing content tests for: - -| Domain | Required cases | -|---|---| -| logistics | rare valid UN/LOCODE-like code; kg/lb and C/F; cross-timezone delivery window; 24:00-like transport time; address PII; late tracking canary; protected shipment ID conflict | -| government | leading-zero district/case IDs; Indian/international grouping; fiscal versus calendar year; multilingual agency names; restricted national ID in notes; mixed legacy encoding; contradictory retention/repair policy | -| education | student IDs; letter/percentage/GPA scales; school-year ambiguity; valid zero score; enrollment date ordering; guardian email/phone and FERPA notes; protected student ID and grade policy conflict | -| insurance | policy/claim IDs; premium/reserve currency conflict; negative reserve review; incident/report date ordering; state transition contradiction; claimant PII and medical loss text; protected policy number conflict | - -- [ ] Run the fixture suite and expect four missing-builder/content failures. - -- [ ] Implement the four deterministic builders using the same 16-row/fixed-reference conventions, complete labels, row cases, schema drift, mixed language/encoding, and PII tail cases. - -- [ ] Add a corpus-level assertion that all required trap categories occur across the eight fixtures and every domain contains all four dispositions. - -- [ ] Rerun `PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q`; expect pass. - -- [ ] Commit: `git add benchmarks/truthbench/fixtures tests/truthbench/test_fixtures.py && git commit -m "feat: complete eight-domain TruthBench corpus"` - -### Task 6: Implement privacy-safe values and exhaustive sink scanning - -**Files:** -- Create: `benchmarks/truthbench/privacy.py` -- Create: `tests/truthbench/test_privacy.py` - -**Interfaces:** Consumes fixture canaries and arbitrary nested sink values (`str`, bytes, mappings, sequences, dataclasses, pandas objects). Produces redacted values/digests and precise leak locations without repeating leaked text. - -- [ ] Write failing mutation tests for literal, case-folded, whitespace-stripped, punctuation-stripped, digit-only, URL-decoded, HTML-unescaped, UTF-8 bytes, NFKC/NFC/NFD, zero-width-removed, and JSON-escaped canary variants. - -```python -@pytest.mark.parametrize("mutate", CANARY_MUTATORS) -def test_scanner_finds_every_normalized_variant(mutate): - scanner = SinkScanner.from_canaries({"crm-email": "tb.person+7@example.invalid"}) - leaks = scanner.scan({"report": [mutate("tb.person+7@example.invalid")]}) - assert [(leak.canary_id, leak.path) for leak in leaks] == [("crm-email", "$.report[0]")] - assert "tb.person" not in repr(leaks) -``` - -- [ ] Run the focused tests; expect import failure. - -- [ ] Implement normalization as named transforms and scan both decoded text and byte hex/escape forms. Use run-scoped HMAC-SHA256 digests and `Leak(canary_id, variant, path)`; never store a matched substring. - -- [ ] Add scanners for exception text, `CleanReport.to_dict()`, action metadata, `coerced_cells`, findings, plan JSON, validation/domain/privacy/Copilot reports, generated code, stdout/stderr, Markdown, HTML, JSON, and failure artifacts. - -- [ ] Verify scanner self-test rejects a result that contains its own canary and passes a correctly redacted sink. - -- [ ] Rerun focused tests; expect pass. - -- [ ] Commit: `git add benchmarks/truthbench/privacy.py tests/truthbench/test_privacy.py && git commit -m "feat: add TruthBench PII leak scanner"` - -### Task 7: Classify every public surface and define the adapter protocol - -**Files:** -- Create: `benchmarks/truthbench/inventory.py` -- Create: `benchmarks/truthbench/surfaces/__init__.py` -- Create: `benchmarks/truthbench/surfaces/base.py` -- Create: `tests/truthbench/test_inventory.py` -- Create: `tests/truthbench/test_surface_adapters.py` - -**Interfaces:** Consumes `freshdata.__all__`, lazy export registries, domain registry, experimental Copilot export, and enterprise CLI parser. Produces one classification for every public name/command: `decision`, `sink`, `explicit-transform`, `data-model`, `configuration`, `registration`, or `out-of-scope-with-reason`. Decision/sink entries must reference an adapter. - -- [ ] Write failing inventory tests comparing the manifest to `fd.__dir__()`, `_ENTERPRISE_EXPORTS`, `_INTEGRATION_EXPORTS`, `_LEARNING_EXPORTS`, `_VALIDATION_EXPORTS`, bundled `domains.available()`, and CLI subcommands. Fail on a new unclassified public name or an adapterless decision/sink. - -- [ ] Run focused tests; expect missing manifest/protocol failures. - -- [ ] Implement frozen `SurfaceSpec` and abstract `SurfaceAdapter.observe(fixture, context) -> SurfaceObservation`. `SurfaceObservation` must carry output frame, raw decisions, audit sinks, trust, backend disclosure, generated code, captured stdout/stderr, and unexpected exception details. - -```python -@dataclass(frozen=True) -class SurfaceSpec: - name: str - version: int - classification: SurfaceClass - adapter: str | None - mutates: bool - deterministic: bool - backend_parity: bool - rationale: str -``` - -- [ ] Explicitly classify low-level `fill_missing`, `remove_outliers`, `resolve_duplicates`, `group_aggregate`, display setters, plugin registration, token vault primitives, and explicit detokenization as caller-directed surfaces; they receive safety/input-output checks but no inferred disposition credit. - -- [ ] Rerun focused tests; expect pass. - -- [ ] Commit: `git add benchmarks/truthbench/inventory.py benchmarks/truthbench/surfaces tests/truthbench/test_inventory.py tests/truthbench/test_surface_adapters.py && git commit -m "feat: inventory FreshData decision surfaces"` - -### Task 8: Implement cleaning, validation, domain, text, and streaming adapters - -**Files:** -- Create: `benchmarks/truthbench/surfaces/cleaning.py` -- Create: `benchmarks/truthbench/surfaces/validation.py` -- Modify: `benchmarks/truthbench/surfaces/__init__.py` -- Modify: `tests/truthbench/test_surface_adapters.py` - -**Interfaces:** Calls public FreshData APIs only. Produces `SurfaceObservation` without grading it. Adapter-specific expected mappings are explicit: mutators repair only `repair`; read-only validators never mutate; `flag` and `review` remain unchanged by default. - -- [ ] Write failing contract tests for `fd.clean`, `Cleaner.clean`, `clean_csv`, `CleanResult`, fluent `pipeline`, `suggest_plan`/`apply_plan`, `validate_fields`/`apply_field_policy`, `fd.validate`/`ValidationSuite`, context compile/validate, bundled domain validators, deterministic semantic assist/review/auto, text clean/lint, and fixed-partition `StreamingCleaner`/`clean_timeseries`. - -- [ ] Run focused tests; expect missing adapters. - -- [ ] Implement adapters with narrow exception capture at the top runner boundary only. Preserve the exact exception type and a scanner-sanitized message. Use public methods for reports and decisions; do not import internal cleaning functions. - -- [ ] Map aggregate actions to cells only when metadata supplies a row, a documented value mapping identifies exact matching cells, or the returned validation/domain rule supplies violation rows. Never award a column-wide action to every labelled cell. - -- [ ] Capture input snapshot, output, `report.actions`, findings, `coerced_cells`, domain reports/repairs, trust fields, plan hashes, recommendations, and rendered representations as separate scan sinks. - -- [ ] For streaming, run identical data through fixed partitions `(8, 8)` and `(5, 5, 6)` and retain batch/rolling/cumulative decisions for later parity comparison. - -- [ ] Rerun focused adapter tests; expect pass. - -- [ ] Commit: `git add benchmarks/truthbench/surfaces tests/truthbench/test_surface_adapters.py && git commit -m "feat: observe core FreshData surfaces"` - -### Task 9: Implement privacy, trust, reporting, export, CLI, and Copilot adapters - -**Files:** -- Create: `benchmarks/truthbench/surfaces/privacy.py` -- Create: `benchmarks/truthbench/surfaces/reporting.py` -- Create: `benchmarks/truthbench/surfaces/copilot.py` -- Modify: `benchmarks/truthbench/surfaces/__init__.py` -- Modify: `tests/truthbench/test_surface_adapters.py` - -**Interfaces:** Calls `detect_pii`, anonymization/privacy policy, trust/quality/compliance/insight, findings/exporters/renderers/CLI, and `experimental.ai_copilot.analyze_dataset(provider=None)`. Produces observations plus every externally visible sink. - -- [ ] Add failing adapter tests for PII detection, anonymization with a fixed test key, privacy policy, k-anonymity, `compute_trust_score`, quality/debt/insight/compliance/stakeholder reports, `to_dict`/`to_frame`/`to_findings`, JSON/Markdown/HTML/Peel/plain rendering, quality-ops/dbt/GX/exception exporters, CLI stdout/stderr, and Copilot provider-free analysis. - -- [ ] Run focused tests; expect missing adapters. - -- [ ] Implement adapters. Copilot must receive `provider=None`; monkeypatch a sentinel provider/network function that fails if called. Record prompt/model context, recommended code, audit, narrative, and all render forms as privacy sinks. - -- [ ] Use a fixed per-test masking secret for deterministic parity, while separately asserting default random masking discloses randomness and never leaks raw PII. - -- [ ] Capture trust on pristine, adversarial, cleaned, and deliberately destructive frames. A destructive control is a same-shape constant/null frame, not an empty frame that bypasses metrics. - -- [ ] Rerun focused tests; expect pass. - -- [ ] Commit: `git add benchmarks/truthbench/surfaces tests/truthbench/test_surface_adapters.py && git commit -m "feat: observe privacy reporting and Copilot surfaces"` - -### Task 10: Add required-backend execution and honest parity checks - -**Files:** -- Create: `benchmarks/truthbench/surfaces/backends.py` -- Create: `tests/truthbench/test_backends_determinism.py` -- Modify: `benchmarks/truthbench/surfaces/__init__.py` - -**Interfaces:** Consumes a fixture and common-native-subset `CleanConfig`. Produces pandas-normalized observations for requested backends `pandas`, `polars`, and `duckdb` with `fallback_policy="error"`, requested/actual backend, fallback events, row identity/order, and report differences. - -- [ ] Write failing tests proving a missing required backend, unexpected fallback, requested/actual mismatch, row reorder, dtype/value divergence, action divergence, or undisclosed backend difference fails parity. Tamper adapters in tests; never rely on an actually missing dependency. - -- [ ] Run focused tests; expect missing backend adapter. - -- [ ] Implement backend preflight with `importlib.metadata.version`, public `fd.clean(..., engine=backend, fallback_policy="error", return_report=True)`, native-to-pandas conversion, and explicit equivalence rules limited to approved representation differences. - -- [ ] Ensure pandas is the reference but not automatically correct: all three outputs are independently scored against gold before parity comparison. - -- [ ] Add extended-profile contracts for Spark/FreshCore. Normal pytest tests adapter/gate behavior using fakes; the extended workflow requires real infrastructure. - -- [ ] Rerun focused tests with installed pandas/Polars/DuckDB; expect pass and no fallback events. - -- [ ] Commit: `git add benchmarks/truthbench/surfaces/backends.py tests/truthbench/test_backends_determinism.py benchmarks/truthbench/surfaces/__init__.py && git commit -m "feat: enforce TruthBench backend parity"` - -### Task 11: Normalize decisions and implement all absolute gates - -**Files:** -- Create: `benchmarks/truthbench/normalize.py` -- Create: `benchmarks/truthbench/gates.py` -- Create: `tests/truthbench/test_normalize.py` -- Create: `tests/truthbench/test_gates.py` - -**Interfaces:** Consumes fixture gold plus a `SurfaceObservation`. Produces one normalized `DecisionRecord` per `(surface, backend, repeat, cell)` and `CaseRecord` per row/schema case, then evaluates gates independently of any baseline. - -- [ ] Write failing normalization tests for exact output/dtype, detected versus mutated, quarantine/review routing, cell-level audit IDs, confidence/rationale/provenance extraction, trust delta, backend disclosure, sensitive values, and non-applicable fields. - -- [ ] Write one independent tamper test for each mandatory gate: - -```python -@pytest.mark.parametrize("mutator, gate", [ - (corrupt_preserve, "valid_value_corruption"), - (modify_protected, "protected_column_modification"), - (leak_canary, "raw_pii_leakage"), - (diverge_backend, "backend_inconsistency"), - (change_repeat, "default_nondeterminism"), - (break_generated_code, "broken_generated_code"), - (remove_high_confidence_explanation, "unexplained_high_confidence"), - (invert_trust, "trust_inversion"), -]) -def test_each_mandatory_gate_fails_independently(passing_run, mutator, gate): - result = evaluate_gates(mutator(passing_run)) - assert failed_gate_names(result) == {gate} -``` - -- [ ] Run focused tests; expect missing normalizer/gates. - -- [ ] Implement surface-aware disposition mapping. Mutators must exactly repair `repair`, preserve `preserve`, and not mutate `flag`/`review`. Validators receive detection credit without mutation. PII adapters are graded only on PII-labelled cells plus false positives. Explicit transforms are graded on requested behavior and protected/privacy invariants. - -- [ ] Implement the eight named gates plus completeness, unexpected exception, required-backend, mutation-audit, review-routing, exact-repair, flag-mutation, input-mutation, and aggregate-consistency gates. High confidence means `>= 0.90`; substantive explanations require non-boilerplate rationale, non-empty rule/model provenance, and a matching audit record. - -- [ ] Make gate evaluation fail closed when records are absent, a surface is unexecuted, schema validation failed, or a run is partial. Baseline comparison may add failures but cannot clear one. - -- [ ] Rerun focused tests; expect pass. - -- [ ] Commit: `git add benchmarks/truthbench/normalize.py benchmarks/truthbench/gates.py tests/truthbench/test_normalize.py tests/truthbench/test_gates.py && git commit -m "feat: enforce TruthBench release gates"` - -### Task 12: Add decision determinism and controlled generated-code execution - -**Files:** -- Create: `benchmarks/truthbench/determinism.py` -- Create: `benchmarks/truthbench/generated_code.py` -- Modify: `tests/truthbench/test_backends_determinism.py` -- Create: `tests/truthbench/test_generated_code.py` - -**Interfaces:** Consumes normalized observations and Copilot code. Produces stable hashes/diffs and a subprocess result that proves AST parse, compilation, offline execution, expected output, and PII safety. - -- [ ] Write failing tests showing hashes ignore duration, peak memory, timestamp, run/lineage IDs, and documented salt bytes but detect decision/order/output/confidence/rationale/audit/trust changes. - -- [ ] Write failing generated-code tests for syntax error, compile error, missing input/output contract, filesystem escape, imports outside the allowlist, `socket`/HTTP access, subprocess creation, raw PII literal, runtime failure, timeout, and wrong cleaned output. - -- [ ] Run the focused tests; expect missing modules. - -- [ ] Implement recursive normalization with an explicit excluded-field set and sorted canonical JSON. Reject callers attempting to exclude a decision-bearing field. - -- [ ] Parse code with `ast.parse`, reject unsafe nodes/imports/calls, compile in-process, then execute in `python -I` with a 10-second timeout, a temporary working directory, environment allowlist, network-denial bootstrap, and serialized synthetic fixture input. Compare the produced frame/report to the adapter contract and scan code/stdout/stderr/artifacts for canaries. - -- [ ] Rerun focused tests; expect pass. - -- [ ] Commit: `git add benchmarks/truthbench/determinism.py benchmarks/truthbench/generated_code.py tests/truthbench/test_backends_determinism.py tests/truthbench/test_generated_code.py && git commit -m "feat: verify deterministic decisions and generated code"` - -### Task 13: Implement runner, deterministic minimizer, CLI, and atomic reports - -**Files:** -- Create: `benchmarks/truthbench/runner.py` -- Create: `benchmarks/truthbench/minimize.py` -- Create: `benchmarks/truthbench/report.py` -- Create: `benchmarks/truthbench/cli.py` -- Create: `benchmarks/truthbench/__main__.py` -- Create: `tests/truthbench/test_minimize.py` -- Create: `tests/truthbench/test_runner_cli.py` - -**Interfaces:** CLI accepts `run --profile release|extended --backends ... --require-backends --repeats N --check`. Runner returns a complete `RunResult`; minimizer consumes one reproducible `GateFailure` and predicate; reporter writes validated, PII-scanned JSON/Markdown atomically. - -- [ ] Write failing reducer tests proving removal order is fixtures/policies, columns, rows, mutations, value/schema/policy simplification, then backend/repeat. The target cell and expected disposition must survive and the same failure ID must still reproduce. - -- [ ] Write failing CLI tests for exact option parsing, unknown backend/profile, missing required backend, partial-run failure, nonzero `--check`, successful atomic replacement, and refusal to write a leaking artifact. - -- [ ] Run focused tests; expect missing runner/CLI/minimizer. - -- [ ] Implement the approved 13-stage runner flow. Use a fresh adapter context per repeat, deterministic fixture/surface ordering, no broad exception suppression, and an infrastructure-failure record when observation cannot complete. - -- [ ] Implement one-minimal reduction with an evaluation budget of 250 calls and a cache keyed by typed fixture/config/surface/backend hash. Generate failure IDs from fixture version, surface, backend, cell/case, gate, and normalized evidence. - -- [ ] Implement `write_atomic()` with same-directory temporary files, `fsync`, schema validation, sink scan, then `os.replace`. Render Markdown only from already-redacted JSON. - -- [ ] Add CLI defaults exactly matching the release command: - -```bash -python -m benchmarks.truthbench run --profile release \ - --backends pandas,polars,duckdb --require-backends --repeats 2 --check -``` - -- [ ] Rerun focused tests; expect pass. - -- [ ] Commit: `git add benchmarks/truthbench tests/truthbench/test_minimize.py tests/truthbench/test_runner_cli.py && git commit -m "feat: run and minimize TruthBench failures"` - -### Task 14: Reproduce and resolve the initial audit hypotheses - -**Files:** -- Create/modify only the regression and source files proven necessary by each reproduction. -- Expected focused files: `tests/regressions/test_truthbench_domain_guard.py`, `tests/regressions/test_truthbench_domain_report.py`, `tests/regressions/test_truthbench_quarantine.py`, `tests/regressions/test_truthbench_semantic_review.py`, `tests/regressions/test_truthbench_findings_audit.py`, `src/freshdata/api.py`, `src/freshdata/report.py`, `src/freshdata/steps/dtypes.py`, `src/freshdata/engine/missing.py`, `src/freshdata/semantic/policy.py`, `src/freshdata/findings.py`. - -**Interfaces:** Consumes real TruthBench failures. Produces minimized privacy-safe reproductions, root-cause notes, permanent regressions, minimal source fixes, and green affected suites. A hypothesis that does not reproduce gets a passing adversarial coverage test and no source change. - -- [ ] Run the release profile without `--check`, capture all failures, and run the minimizer for each: - -```bash -PYTHONPATH=src python -m benchmarks.truthbench run --profile release \ - --backends pandas,polars,duckdb --require-backends --repeats 2 -``` - -- [ ] For domain protection, add a focused test where a compiled policy marks a repairable domain column immutable. Expect byte-identical output or `ProtectedColumnError`; reproduce before changing `api.py`. If reproduced, snapshot the resolved protected columns before `run_domain`, verify the repaired frame afterward, and record the guard action. - -- [ ] For post-domain report state, assert `rows_after`, `cols_after`, `memory_after`, and `missing_after` describe the returned domain-repaired frame. If reproduced, centralize final metric refresh and call it after `_fold_domain_outcome`. - -- [ ] For late coerced cells, build 1,020 parse casualties where a row after index 1,000 would otherwise be imputed. If reproduced, add a non-serialized `_coerced_rows: dict[str, set[Any]]` to `CleanReport`, populate every lost index in `_record_coerced`, keep `coerced_cells` recovery values capped, and make `_quarantined_rows` use the complete internal index set. - -- [ ] For semantic review, inject a high-confidence low-risk proposal from `embedding`, `profile`, `memory`, and `plugin:*` backends. If any applies in review mode, restrict review-mode auto-application to deterministic built-in provenance; keep non-default proposals suggested with human review. Preserve `auto` behavior subject to its existing safety gates. - -- [ ] For finding audit projection, assert a medium/high-risk action retains confidence, rationale, model/rule ID, status, reversible, human-review, and safe metadata in `QualityFinding.extra`. If lost, pass complete action dictionaries from `CleanReport.to_findings()` and explicitly copy those safe audit fields in `findings_from_dict()`. - -- [ ] For every reproduced defect, follow this exact loop separately: run the failing focused test; save minimized redacted failure JSON; implement one root-cause fix; run the focused test; run the affected module suite; rerun its TruthBench case; commit with a concise `fix:` message naming the observed root cause. - -- [ ] Exercise the remaining hypotheses—Gauntlet row mapping, domain cell evidence, audit over-attribution, raw recovery leakage, random salt normalization, and tail-only plan drift—through TruthBench. Fix FreshData only if the new exact benchmark reproduces a public-contract violation; otherwise retain the adversarial test as coverage. - -- [ ] After each fix, scan the patch for broad catches/skips/changed gold: - -```bash -git diff origin/main -- tests src benchmarks/truthbench | rg "pytest\.skip|xfail|except Exception|COERCED_CELLS_CAP|expected|Disposition" -``` - -- [ ] Commit each validated fix independently; do not combine unrelated root causes. - -### Task 15: Commit benchmark artifacts and document every result - -**Files:** -- Create: `benchmarks/truthbench/results/README.md` -- Create: `benchmarks/truthbench/results/baseline.json` -- Create: `benchmarks/truthbench/results/latest.json` -- Create: `benchmarks/truthbench/results/latest.md` -- Create/update: `benchmarks/truthbench/results/failures/*.json` -- Create: `docs/truthbench.md` -- Modify: `README.md` - -**Interfaces:** Consumes the clean release run. Produces versioned, schema-valid, PII-free evidence and user documentation. Baseline detects regressions but cannot waive an absolute gate. - -- [ ] Add failing tests that committed artifacts validate, match current fixture hashes/surface manifest, contain all eight domain names and all gate results, contain no canary variants, and do not claim success for unresolved failures. - -- [ ] Run those tests; expect missing artifact failures. - -- [ ] Run the full release profile with `--check`; write `latest.json`/`latest.md`, then intentionally copy the verified result to `baseline.json` only after every absolute gate passes. - -- [ ] Document architecture, disposition meanings, surface mapping, exact comparator, privacy model, failure reproduction, minimization, baseline update policy, local commands, and the explicit prohibition on LLMs in the runtime path. - -- [ ] In `latest.md`, list implemented files, each discovered failure and root cause, each fix/regression, exact commands, per-domain/per-surface/backend results, every gate's evidence counts, remaining limitations, and the two pre-existing timing flakes with their passing exact reruns. - -- [ ] Rerun artifact tests and `rg` every planted canary across committed result/docs paths; expect zero matches outside fixture source and scanner test data. - -- [ ] Commit: `git add benchmarks/truthbench/results docs/truthbench.md README.md tests/truthbench && git commit -m "docs: publish TruthBench release evidence"` - -### Task 16: Make TruthBench a mandatory PR and production release gate - -**Files:** -- Modify: `Makefile` -- Modify: `.github/workflows/ci.yml` -- Modify: `.github/workflows/release.yml` -- Create: `.github/workflows/truthbench-extended.yml` -- Modify: `pyproject.toml` only if a dedicated benchmark extra is required after verification. - -**Interfaces:** CI/release consumes the repository and required dependencies. Produces a hard pass/fail before packaging/publication plus scheduled extended artifacts. - -- [ ] Add failing workflow/Makefile contract tests asserting exact required command, no `continue-on-error`, no skip-on-missing-backend branch, and TruthBench execution before `python -m build`/publication. - -- [ ] Add targets: - -```make -.PHONY: truthbench truthbench-release truthbench-extended -truthbench: - PYTHONPATH=src python -m benchmarks.truthbench run --profile release --backends pandas,polars,duckdb --require-backends --repeats 2 -truthbench-release: - PYTHONPATH=src python -m benchmarks.truthbench run --profile release --backends pandas,polars,duckdb --require-backends --repeats 2 --check -truthbench-extended: - PYTHONPATH=src python -m benchmarks.truthbench run --profile extended --backends pandas,polars,duckdb,spark,freshcore --require-backends --repeats 2 --check -``` - -- [ ] Install dev/out-of-core dependencies in PR/release jobs and add `make truthbench-release` after fast pytest and before build. The scheduled extended job provisions JVM/native infrastructure and uploads `latest.json`, `latest.md`, and minimized failures on success or failure. - -- [ ] Run workflow syntax/contract tests plus `make -n truthbench-release`; expect the exact command. - -- [ ] Commit: `git add Makefile .github/workflows tests pyproject.toml && git commit -m "ci: require TruthBench release gates"` - -### Task 17: Final verification and self-review - -**Files:** -- Review all changes from `origin/main...HEAD`. -- Update `benchmarks/truthbench/results/latest.json` and `latest.md` only if verification evidence changed. - -**Interfaces:** Consumes the finished branch. Produces reproducible evidence that the specification and every release gate are satisfied without test weakening. - -- [ ] Run focused TruthBench tests: - -```bash -PYTHONPATH=src python -m pytest tests/truthbench tests/regressions/test_truthbench_*.py -q -``` - -Expected: all pass, no skips. - -- [ ] Run the mandatory release benchmark: - -```bash -PYTHONPATH=src python -m benchmarks.truthbench run --profile release \ - --backends pandas,polars,duckdb --require-backends --repeats 2 --check -``` - -Expected: exit 0; all eight mandatory and additional completeness gates pass; no backend fallback; zero raw-PII findings. - -- [ ] Run existing correctness systems unchanged: - -```bash -PYTHONPATH=src python -m benchmarks.gauntlet run --check -PYTHONPATH=src python benchmarks/public_benchmark.py -PYTHONPATH=src python -m pytest -m "not online and not large" -``` - -Expected: Gauntlet gates pass, public/CleanBench benchmark passes, pytest passes apart from no accepted failures. If the two recorded timing tests flake, rerun their exact node IDs and report both outputs; do not alter thresholds. - -- [ ] Run static/package checks: - -```bash -ruff check . -mypy src/freshdata -python -m build -python -m twine check dist/* -``` - -Expected: all exit 0. - -- [ ] Perform specification traceability review: map every approved design section and acceptance criterion to an implemented file plus a passing test/result field. - -- [ ] Scan for incomplete implementation and policy violations: - -```bash -rg -n "TODO|FIXME|NotImplemented|pass$|pytest\.skip|xfail|continue-on-error" benchmarks/truthbench tests/truthbench tests/regressions .github/workflows -rg -n "openai|anthropic|provider=|requests\.|httpx\.|urllib" benchmarks/truthbench src/freshdata -``` - -Expected: no placeholders, hidden skips, release `continue-on-error`, or network/model calls in the benchmark/default path; the Copilot adapter contains only the explicit `provider=None` assertion and network-denial test harness. - -- [ ] Review `git diff --check`, `git status --short`, and the full commit list. Confirm the original checkout remains unchanged. - -- [ ] If verification changed result evidence, regenerate it, rerun schema/PII checks, and commit `test: finalize TruthBench release evidence`. - -- [ ] Prepare the final handoff with implemented files, discovered failures, fixes, commands and outputs, benchmark totals, limitations, and per-gate evidence. diff --git a/docs/superpowers/specs/2026-07-11-freshdata-confirmed-performance-bottlenecks-design.md b/docs/superpowers/specs/2026-07-11-freshdata-confirmed-performance-bottlenecks-design.md deleted file mode 100644 index 745cc00..0000000 --- a/docs/superpowers/specs/2026-07-11-freshdata-confirmed-performance-bottlenecks-design.md +++ /dev/null @@ -1,262 +0,0 @@ -# FreshData Evidence-Gated Semantic Optimization Design - -**Status:** Approved in conversation on 2026-07-11 -**Scope:** Phase 2 design only; no production change is authorized by this document alone - -This design follows the -[approved performance investigation design](./2026-07-11-freshdata-performance-scalability-design.md) -and the completed [performance investigation](../../performance-investigation.md). -It addresses the investigation's sole optimization candidate without treating -that candidate as a confirmed root cause or promising a performance gain. - -## Decision - -Measure exact repeated semantic predicate/parser arguments first. Only if that -measurement demonstrates a material cacheable opportunity may implementation -add exact, request-local result reuse inside semantic-context construction. The -change is accepted only if it preserves the complete behavioral contract and -passes the end-to-end performance and memory gates below. Otherwise it is -rejected or reverted; the scope does not expand to another optimization. - -## Measured Evidence - -The only Phase 2 candidate is baseline case `9ea9cb03dfc114c5`: - -- 100,000 rows, medium width, mixed data, pandas input/output; -- `semantic_mode="assist"`, `verbose=False`, `return_report=True`, seed 42; -- one warm-up and five measured repetitions; -- profile total `22.015065380` seconds; -- semantic-stage fraction `0.13104697297973694` - (`13.104697297973694%`); -- `run_semantic`: one call, `6.300034834000001` seconds cumulative; -- `build_semantic_context`: one call, approximately `6.184` seconds cumulative; -- `_build_info`: 32 calls, approximately `5.200` seconds cumulative; -- `_share`: 192 calls, approximately `4.232` seconds cumulative. - -The capped self-time samples include `is_plain_number` at approximately -`0.718468278` seconds over 192,602 calls, `looks_like_date_value` at -approximately `0.387268678` seconds over 50,956 calls, and the boolean -predicate lambda in `semantic/context.py` at approximately `0.327692425` -seconds over 192,602 calls. - -### Evidence limits - -These measurements identify a candidate and hot functions, not a cause. The -profiler lists are capped and non-exhaustive. `_build_info` already applies -`pd.unique` within every bounded column sample, so reuse across columns or -context builds cannot be assumed. Exact repetition must be measured. - -Every other investigated hypothesis was rejected or had insufficient evidence. -The required MissForest profile was interrupted after approximately two -CPU-bound hours and produced no JSON, so no MissForest conclusion is available. - -## Compatibility Contract - -The optimized and current paths must be indistinguishable across: - -- public signatures, configuration fields, return types, and deterministic output; -- values, shape, row/column order, index values, names, and types; -- exact dtypes and input-mutation behavior; -- warning and exception type, message, and ordering; -- actions and action ordering, counts, rationales, risks, confidence, - recommendations, and serialized reports; -- protected identifier, target, preserve, and PII behavior and privacy policy; -- semantic modes `off`, `assist`, `review`, and `auto`; -- memory/profile replay and native/fallback behavior. - -Sampling, `pd.unique`, thresholds, classification, proposal resolution, -replacement order, and audit/report construction remain unchanged. - -## Current Data Flow - -```text -run_semantic - -> build_semantic_context - -> build_contexts - -> for each column, _build_info - -> bounded non-null sample - -> pd.unique within that sample - -> semantic predicates/parsers - -> SemanticColumnInfo - -> SemanticContext - -> gather proposals - -> calibrate proposals and decide - -> build replacement map - -> record every decision - -> apply replacements -``` - -The only proposed boundary is the single `build_semantic_context` invocation. -The cache must not reach proposal gathering, replacement resolution, or report -construction. - -## Proposed Design - -### 1. RED discovery gate - -Add development-only instrumentation around the named pure predicate/parser -calls made by `_build_info`. On the representative workload, record per -operation: - -- total calls and the exact argument sequence; -- cache-eligible and bypassed calls; -- unique keys, repeated keys, theoretical hits, and hit rate; -- repeated-evaluation time sufficient to judge whether a 10% end-to-end win is - feasible. - -Reset the discovery counters and simulated cache at every -`build_semantic_context` entry. Compute unique keys and theoretical hits only -within that one build, then emit and aggregate the completed per-build metrics. -Aggregation may sum per-build counts but must not deduplicate across builds or -carry simulated entries forward. Repetition between warm-up, measured, memory, -or profile runs—or between any other context builds—never counts as a cacheable -hit. - -The prospective key is `(predicate identity, exact input type, exact input -value)`. Measurement must use the same eligibility and key rules as the proposed -implementation. Call count alone is not sufficient: repeated eligible work must -be material enough to make the final acceptance threshold plausible. Record the -predeclared calculation and result. If it does not establish that opportunity, -stop with no production change. - -After discovery passes, add a failing focused test proving that the current path -re-evaluates a repeated eligible key within one context build. The test must also -capture the uncached dataframe/context and complete report/audit fingerprint for -later differential comparison. - -Instrumentation belongs in focused tests or benchmark tooling and adds no -default production overhead. - -#### Candidate universe and exact input eligibility - -Discovery instruments exactly these seven operations and input types: - -- `is_plain_number`: exact built-in `str`, `int`, `float`, or `bool`; -- `parse_number_words`: exact built-in `str`; -- `parse_boolean`: the actual post-`str(v)` argument, exact built-in `str`; -- `parse_currency`: exact built-in `str`; -- `parse_unit`: exact built-in `str`; -- the email-shape predicate `bool(_EMAIL_VALUE.match(v.strip()))`: exact built-in - `str`; -- `looks_like_date_value`: exact built-in `str`. - -All non-allowlisted exact types bypass before any hashing or equality operation. -Subclasses, NumPy scalars, and user-defined values are never cached. Discovery -measures all seven operations; production may enable only the subset that -discovery proves material and that the implementation plan names explicitly. No -other operation is eligible. - -### 2. Minimal GREEN change - -Create one private cache when `build_semantic_context` starts and share it only -with that invocation's `_build_info` calls. Destroy it when the build returns or -raises. Do not add a public parameter or configuration switch. Do not add a size -cap: the one-build lifetime and the memory acceptance gate are the controlling -bounds, and the evidence does not justify another policy. - -Evaluate only explicitly named, pure deterministic operations through the -request-local helper. Use the underlying parser/predicate identity, not an -ephemeral lambda identity. For transformations such as `parse_boolean(str(v))`, -perform the existing `str(v)` conversion in its existing order, then key the -parser by the exact string it receives. - -Include both `type(value)` and `value` in every key so `1`, `True`, and -numeric-equivalent values of different Python types cannot collide. Apply the -exact operation/type allowlist above before hashing or equality; every other -value bypasses directly. - -Call the original operation on a miss. Store a result only after successful -return and only when its result contract is immutable. The current candidate -results are `None`, booleans, integers, floats, and the immutable `(float, str)` -unit tuple. They remain internal to context scoring, so object identity is not -exposed. Mutable results are never cached. - -Do not cache exceptions. Do not catch, translate, reorder, or suppress a parser -exception. A bypass calls the original operation at the same logical point. A -miss that raises propagates the same exception, and a second context build starts -with an empty cache. - -### 3. Refactor and verification gate - -Keep cache policy, lookup, and result eligibility in one small private unit; -keep `_build_info` responsible for unchanged sampling and scoring. Refactoring -must not generalize the cache beyond semantic-context construction. - -Run differential tests against the captured uncached reference and the full -compatibility matrix. Re-run the representative benchmark and profiler only -after correctness passes. - -## Test Design - -The later implementation plan must use strict TDD: - -1. RED: prove measured repetition, then fail on avoidable underlying calls while - capturing the complete reference behavior. -2. GREEN: add only the request-local exact-result cache needed to pass. -3. REFACTOR: isolate the private helper without changing behavior or scope. -4. VERIFY: run differential, regression, performance, memory, and profile gates. - -Focused cache tests cover a hit within one build, no reuse across builds, -predicate/type/value key separation, safe bypass, immutable result reuse, and -unchanged exception propagation. Broader differential coverage includes: - -- normal and repeated inputs; mixed Python types and unhashable values; -- nullable and object dtypes and supported duplicate labels; -- protected columns, semantic hints, policies, and all semantic modes; -- report enabled/disabled, empty, single-row, and all-null inputs; -- native/pandas parity and native fallback; -- memory and profile replay. - -Differential fingerprints compare the complete dataframe surface—values, shape, -ordering, index, and exact dtypes—and the complete report/audit surface, -including ordered actions, counts, rationales, risks, confidence, -recommendations, metadata, warnings, and serialization. - -## Performance Acceptance Gate - -Measure before and after in the same recorded environment using the primary -case: 100k rows, medium width, semantic assist, `verbose=False`, report enabled, -pandas input/output, seed 42, one warm-up, and five measured repetitions. - -Acceptance requires all of the following: - -- at least 10% median wall-time improvement; -- improvement greater than twice the larger before/after coefficient of - variation, using the repository's comparison semantics; -- exact dataframe and report/audit fingerprint equality on every measured run; -- no material peak-RSS or Python-allocation-peak regression; -- no meaningful regression in the 100k/medium/report=true default and aggressive - controls under the repository's precise comparison semantics; -- a new semantic profile showing time removed from the targeted functions rather - than shifted to another stage. - -The evidence record includes environment, commit, dirty state, exact commands, -individual repetitions, medians, coefficients of variation, peak RSS, Python -allocation peak, fingerprints, profile deltas, and accepted/rejected outcome. - -## Failure, Rollout, and Rejection - -There is no feature flag or staged public API. This is an internal optimization -that may land only after every compatibility and acceptance gate passes. - -Reject or revert the production change if repetition is not material, any -behavior differs, the runtime threshold is missed, memory or controls regress, -or profiling shows displaced rather than removed work. Preserve the negative -measurement as evidence and do not broaden the implementation to recover a win. - -## Rejected Alternatives - -- Module, global, persistent `functools.lru_cache`, or cross-request caching: - rejected for state leakage, unbounded retention, and semantic risk. -- Vectorizing or rewriting parsers/classifiers: rejected as excessive behavioral - risk before exact reuse is measured. -- MissForest optimization: rejected because no completed profile exists. -- Backend rewrites and correlation, copy, dtype, reporting, uniqueness, or - null-scan work: outside the sole measured Phase 2 candidate. - -## Non-Goals - -This phase does not change semantic decisions, sampling, configuration, public -interfaces, backend architecture, dependencies, reports, or documentation -claims. It does not create a general memoization framework or optimize any -unmeasured subsystem. diff --git a/docs/superpowers/specs/2026-07-11-freshdata-performance-scalability-design.md b/docs/superpowers/specs/2026-07-11-freshdata-performance-scalability-design.md deleted file mode 100644 index d175898..0000000 --- a/docs/superpowers/specs/2026-07-11-freshdata-performance-scalability-design.md +++ /dev/null @@ -1,360 +0,0 @@ -# FreshData Performance and Scalability Investigation Design - -**Status:** Approved in conversation on 2026-07-11 - -**Baseline:** Git commit `6f6c2fe` on branch -`fix/public-exports-and-inf-outliers-jwd` - -## Objective - -Measure, explain, and reduce FreshData's verified runtime and memory bottlenecks -without changing its public API, cleaning decisions, safety guarantees, audit -trail, context policies, validation behavior, or supported backends. The work -must independently reproduce or disprove each performance hypothesis, retain -only evidence-backed optimizations, and publish the commands and results needed -to reproduce every claim. - -An optimization is a verified win only when it improves the relevant median -runtime or peak-memory measurement by at least 10%, the improvement is greater -than twice the observed run-to-run variability, and it introduces no meaningful -regression in another primary workload. - -## Compatibility Constraints - -- Python support remains `>=3.9`, including the project's tested Python - 3.9-3.13 matrix. -- pandas support remains `>=1.5,<3`; NumPy remains `>=1.21`. -- The default cleaning strategy remains `balanced`, with the current - representation-repair defaults unchanged. -- Public function and class signatures, configuration fields, return types, - warnings, exceptions, index semantics, and audit/report contracts remain - compatible. -- No unrelated runtime dependency is introduced. Profiling additions use the - standard library and already-declared benchmark/development dependencies. - -## Current Architecture and Execution Flow - -The public cleaning path is: - -```text -fd.clean - -> normalize API options and validate contracts/context/memory/profile inputs - -> select the pandas or native execution path - -> initialize CleanReport and preserve-original handling - -> representation repair - column names - strings and sentinels - empty rows and columns - dtype repair - optional constant-column removal - duplicate handling - optional semantic repair - -> decision engine - build per-column contexts - build reusable engine statistics/correlations - automatic missing-value decisions - automatic outlier decisions - -> explicit imputation and outlier overrides - -> optional memory optimization - -> protected-column verification and index handling - -> finalize report - -> wrap or convert the result -``` - -The pandas implementation in `src/freshdata/cleaner.py` is the behavioral -reference. `src/freshdata/engine/context.py` profiles roles and statistics, -`src/freshdata/engine/cache.py` shares artifacts between automatic missing and -outlier processing, and `src/freshdata/steps/` implements representation-level -operations. Native engines in `src/freshdata/execution/backends/` reproduce a -documented subset and fall back to pandas when required. Every path must retain -the existing `CleanReport` contract. - -`return_report=False` does not mean that report construction can be removed: -the pandas result remains a `CleanResult` with an embedded report and -`Cleaner.report_` is always populated. Optimization may defer or avoid only -internal work proven unnecessary while preserving those behaviors exactly. - -## Chosen Approach - -Use an evidence-gated sequence: - -1. Establish an immutable baseline and a reproducible benchmark/profiling - harness. -2. Profile the complete pandas pipeline and separate core cleaning, reporting, - optional ML/semantic work, conversion, and backend costs. -3. Implement one measured optimization at a time behind equivalence tests. -4. Re-run the affected benchmark slice after each change and discard changes - that do not exceed the noise threshold. -5. Expand CI-safe regression checks and scheduled/manual large-data coverage. -6. Evaluate native and out-of-core options from measurements rather than begin - with an architectural rewrite. -7. Correct documentation and assemble the final evidence package. - -This approach was selected over a broad refactor, which would obscure causal -attribution, and a backend-first rewrite, which would add compatibility risk -before the reference pandas bottlenecks are known. - -## Workstream Boundaries - -### 1. Baseline and Profiling Infrastructure - -Create a focused `benchmarks/performance/` package with independent modules for -dataset generation, run configuration, subprocess measurement, profiling, -result validation, comparison, and Markdown rendering. Keep it separate from -production imports. Reuse existing benchmark utilities only where their result -schema and measurement semantics match this investigation. - -The authoritative output is versioned JSON. Generated Markdown is a view of -that JSON. Each result records: - -- Git commit and dirty-state indicator. -- FreshData, Python, pandas, NumPy, and optional-backend versions. -- Operating system, CPU, logical/physical core count, and available RAM. -- Dataset seed, shape, width profile, and column-family counts. -- Full cleaning configuration, backend, output format, and report flag. -- Warm-up count, measured repetitions, and exact command. -- Per-run wall time, median, minimum, maximum, standard deviation, coefficient - of variation, and rows per second. -- Peak RSS increase, Python allocation peak, input bytes, and input-to-peak - ratio. -- Equivalent pandas baseline time, FreshData slowdown ratio, and before/after - FreshData improvement ratio where the comparison is semantically valid. -- Completion, skip, failure, timeout, or memory-exhaustion status. - -The command-line interface accepts configurable row count, column count or -width profile, dataset type, cleaning configuration, repetition count, backend, -report generation, output format, seed, timeout, and JSON/Markdown destination. - -### 2. Pandas Pipeline Optimization - -Treat the following as hypotheses until profiling confirms them: - -- Repeated null, non-null, uniqueness, mode, dtype, role, and shape scans can - be reused through the engine cache. -- The full numeric correlation matrix is built when no consumer can use it. -- Missing and outlier dispatch rescans columns despite exact context data. -- String, sentinel, dtype, or numeric operations perform avoidable copies or - conversions. -- Report finalization repeats full-frame work that can be safely shared. -- A precomputed execution plan can reduce dispatch overhead for repeated - compatible transformations. - -Only exact reuse is allowed. If a cached value could be stale after an earlier -stage changes a column, the current computation remains authoritative. -Correlation work may be skipped only when it is provably unused, including -cases such as balanced mode, explicit imputation, frames above the existing -10,000-row KNN limit, or the absence of an eligible medium-missing numeric -target. When correlations are needed, a targeted calculation is acceptable -only if it produces pandas-equivalent values. - -Batching is permitted only for compatible columns and must retain column order, -dtypes, action ordering, counts, rationales, risks, confidence values, warnings, -and exceptions. Copy reductions must preserve `preserve_original`, pandas -copy-on-write compatibility, protected-column snapshots, and documented input -mutation behavior. - -### 3. Correctness and Regression Coverage - -Add differential tests that compare the optimized implementation with baseline -behavior for values, shape, column order, index values/names, exact dtypes, -input mutation, actions and action order, counts, rationale, risk, confidence, -warnings, recommendations, fallback events, and serialized reports. - -Coverage includes: - -- Normal mixed frames, empty frames, and single-row frames. -- All-null and constant columns. -- Boolean and nullable-boolean columns. -- Nullable integer columns and ordinary integer columns. -- Float32, float64, finite values, NaN, and positive/negative infinity. -- Categorical columns, including missing categories. -- Datetime and timezone-aware columns. -- Mixed object payloads and unhashable values. -- Identifiers, targets, context-protected columns, and PII-sensitive fields. -- Duplicate rows, supported duplicate labels, and duplicate-resolution modes. -- Standard indexes, named indexes, DatetimeIndex, and MultiIndex where - supported. -- Narrow, medium, wide, and large frames. -- Report-enabled and report-disabled calls. -- Conservative, balanced, aggressive, explicit imputation/outlier, semantic, - context-policy, memory/profile replay, and native-fallback paths. -- Existing exception types, validation order, and warning text/categories. - -CI performance tests assert stable structural properties rather than tiny wall -time differences. Examples include proving that correlations are not computed -when unused, exact cached counts are reused, and disabled features do not invoke -their stages. Runtime and peak-memory thresholds belong in scheduled or manual -performance jobs. - -### 4. Backend and Out-of-Core Evaluation - -Measure pandas, Polars, DuckDB, Spark, and FreshCore where their optional -dependencies and runtime requirements are available. For every relevant step, -record whether execution is native, materialized, converted, or delegated to -pandas, including conversion time, peak memory, fallback events, and audit -differences. - -Produce evidence-based recommendations for: - -- A native Polars lazy execution engine. -- DuckDB larger-than-memory processing. -- Chunked pandas execution. -- Rust/FreshCore acceleration of verified hot loops. - -Do not implement a large backend rewrite unless optimized pandas measurements -still miss the scalability targets and the proposed native path can preserve -the public behavior and audit contract. Otherwise provide an implementation -proposal with boundaries, milestones, compatibility requirements, and -benchmark targets. - -### 5. Documentation and Verification - -Publish `docs/performance-investigation.md` as the evidence-backed report and -update `README.md`, benchmark documentation, backend documentation, -limitations, and production-readiness claims where measurements contradict or -qualify phrases such as "fast", "vectorized", "scalable", or "memory -efficient". - -The documentation states measured performance ranges, known scalability and -correlation-cost limits, memory considerations, report/non-report behavior, -recommended large-data configurations, and the status of optional fast or -backend-native paths. - -No benchmark result may be presented without its environment and command. Poor -results and unresolved limitations remain visible. - -## Benchmark Matrix - -The deterministic mixed-schema generator covers numeric, categorical, string, -nullable, datetime, timezone-aware, identifier, target, duplicate, missing, -outlier, and high-cardinality data. - -| Dimension | Values | -|---|---| -| Rows | 10,000; 100,000; 500,000; 1,000,000 | -| Width | narrow: 8; medium: 32; wide: 128 columns | -| Cleaning | default balanced; conservative; representation features disabled; statistical features disabled; explicit imputation/outliers; optional ML/semantic | -| Report flag | `return_report=True`; `return_report=False` | -| Backend | pandas; installed Polars, DuckDB, Spark, FreshCore paths | -| Output | pandas and supported eager/lazy native output formats | -| Repetitions | one warm-up, then five measured runs | - -Equivalent pandas baselines are used only for comparable component workloads; -they must implement the same selected transformation and may not be described -as equivalent to FreshData's full decision and audit pipeline. - -## Profiling Design - -Profiling is development-only and adds no default production overhead: - -- `cProfile` attributes function-level runtime. -- `tracemalloc` snapshots identify allocation-heavy files and lines. -- Stage timers isolate context construction, engine-cache construction, - correlation, missing processing, outlier processing, role inference, dtype - repair, duplicate handling, audit generation, final report generation, - optional ML/semantic work, backend conversion, and materialization. -- Controlled instrumentation counts observed calls to `DataFrame.copy`, - `Series.copy`, and selected conversion methods. -- Scan counters observe null counts, uniqueness calculations, correlations, - dtype conversions, and role inference. - -Copy-call counts are explicitly labelled observations and are not treated as a -complete measure of physical pandas buffer copies. Peak RSS and allocation -measurements run in isolated subprocesses so previous runs and allocator residue -cannot contaminate results. - -## Error Handling - -Benchmark failures are data, not successes. The harness writes a failure record -with the command, environment, exception type/message, and partial measurements. -Unavailable optional backends are marked skipped with the exact missing -requirement. Timeouts and memory exhaustion remain visible. - -Production behavior is stricter: optimizations preserve existing exception -types, messages where asserted, validation order, warnings, and fallback -events. Cached or optimized logic falls back to the current computation when -the required invariant cannot be proven. A change that cannot demonstrate -equivalence is rejected or documented as a future proposal. - -## Change Gate - -Every production optimization follows this sequence: - -```text -write a failing equivalence or structural-performance test - -> verify the expected failure - -> implement the smallest internal change - -> pass focused correctness tests - -> compare against baseline behavior - -> run the relevant benchmark slice - -> retain only if the gain exceeds the noise threshold -``` - -A local gain that causes a regression elsewhere must be narrowed behind a -proven eligibility predicate or removed. - -## CI and Scheduled Workflows - -Pull-request CI remains bounded and deterministic. It validates the benchmark -schema, generator determinism, report rendering, structural performance -properties, and representative small/medium equivalence cases. - -A scheduled and manually dispatchable workflow runs the 100,000 through -1,000,000-row profiles, publishes JSON/Markdown artifacts, and includes enough -metadata to compare runs only on compatible environments. It supplements rather -than replaces the existing CleanBench and performance-regression workflows. - -## Required Deliverables - -`docs/performance-investigation.md` contains these explicit sections: - -1. Executive summary. -2. Reproduction commands. -3. Baseline benchmark table. -4. Profiling findings with functions, files, and lines. -5. Confirmed root causes. -6. Rejected hypotheses. -7. Files and functions changed. -8. Explanation of every optimization. -9. Behavioral compatibility analysis. -10. Tests added or changed. -11. Before-and-after benchmark table. -12. Peak-memory comparison. -13. Remaining bottlenecks. -14. Backend and out-of-core recommendations. -15. Documentation corrections. -16. Risks and trade-offs. -17. Exact verification commands and results. - -Each important change separately states its problem, evidence, implementation, -correctness protection, measured performance impact, and remaining risk. - -## Final Verification - -Run and record all repository-supported gates: - -- Complete pytest suite, including online/large tests where their fixtures are - available. -- Ruff linting and formatting checks. -- mypy type checking. -- Repository-supported security checks. -- Strict MkDocs build. -- Source and wheel build plus `twine check`. -- Existing benchmark and CleanBench suites. -- New complete benchmark matrix and profiler. -- Available Polars, DuckDB, Spark, and FreshCore checks. - -The final completion audit maps every objective requirement to authoritative -evidence. A missing optional service, dependency, fixture, or runtime is -reported explicitly and prevents an unsupported claim; it is never silently -omitted. - -## Non-Goals - -- Changing public signatures or defaults. -- Weakening validation, auditability, safety, context policy, PII, identifier, - or target protections. -- Hiding report work by removing the embedded report contract. -- Adding unrelated runtime dependencies. -- Replacing pandas with a native backend without measured justification. -- Publishing marketing claims that are not supported by recorded results. diff --git a/docs/superpowers/specs/2026-07-14-freshdata-truthbench-design.md b/docs/superpowers/specs/2026-07-14-freshdata-truthbench-design.md deleted file mode 100644 index f2e2bee..0000000 --- a/docs/superpowers/specs/2026-07-14-freshdata-truthbench-design.md +++ /dev/null @@ -1,498 +0,0 @@ -# FreshData TruthBench Design - -**Date:** 2026-07-14 -**Target:** `origin/main` at `a1da862` -**Implementation branch:** `feature/truthbench-jwd` -**Status:** Approved for implementation planning - -## Purpose - -FreshData TruthBench is a deterministic semantic red-team and regression system. -It continuously searches for cases where a public FreshData data-quality surface -makes an unsafe, incorrect, inconsistent, or unexplained decision, then converts -each reproduced failure into a minimized case and a permanent regression test. - -TruthBench is test infrastructure, not a cleaning engine. No LLM, teacher model, -provider hook, or generative runtime may participate in FreshData's default -cleaning path or in a TruthBench release run. An LLM may only help humans author -adversarial seed cases, criticize results, and analyze reproduced failures outside -the executable benchmark. - -## Repository Context - -TruthBench is additive to the existing quality layers: - -- CleanBench measures frame-level repair fidelity, calibration, preservation, - profile replay, privacy, and performance. -- The Validation Gauntlet measures labelled dispositions across five fixtures, - but uses partial cell labels and a deliberately loose value comparator. -- The enterprise fixture benchmark measures repair, preservation, trust - monotonicity, reporting completeness, and scale. -- Golden, real-world, online, domain, integration, backend, Copilot, privacy, and - generated-code pytest suites protect individual public contracts. -- The performance investigation provides subprocess-isolated scalability - evidence and is not a semantic correctness oracle. - -TruthBench does not replace, rename, weaken, or rebaseline these systems. It adds -the strict cross-surface cell-level contract they do not currently provide. - -The isolated baseline run on the target commit produced 3,692 passes, 11 skips, -12 deselections, and 93.66% coverage. Two existing throughput assertions failed -during the seven-minute full run: -`balanced-abalone` missed its allowed floor by approximately 0.004%, and -`aggressive-gapminder` was slower than its allowed floor. Both exact tests passed -immediately when rerun with coverage enabled. These are recorded as pre-existing, -non-reproducible environment timing flakes. TruthBench must not relax, skip, or -change either expectation. - -## Chosen Approach - -TruthBench will live in a dedicated `benchmarks/truthbench/` package. It may reuse -stable public APIs and small general-purpose helpers from existing benchmarks, -but its oracle, exact comparator, result schema, gates, surface adapters, failure -reducer, and eight-domain corpus are independent. - -This is preferred over extending the Validation Gauntlet because the Gauntlet's -loose canonical equality and partial labels are part of its historical contract. -It is preferred over a CleanBench T6 track because CleanBench is organized around -aggregate frame metrics rather than one record per cell and surface. - -## Scope - -### In scope - -TruthBench will: - -1. Generate deterministic gold-labelled fixtures for finance, healthcare, - retail, CRM, logistics, government, education, and insurance. -2. Assign exactly one disposition to every test cell: - `preserve`, `repair`, `flag`, or `review`. -3. Exercise every public FreshData surface that makes, applies, transports, - renders, serializes, or gates a data-quality decision. -4. Normalize heterogeneous surface outputs into a common decision record. -5. Compare actual decision, expected decision, output value, confidence, - rationale, audit completeness, trust-score change, backend consistency, and - determinism. -6. Detect raw PII leakage into reports, logs, model context, generated code, - rendered output, CLI output, and committed benchmark artifacts. -7. Minimize every reproduced failure to a stable, privacy-safe case. -8. Enforce absolute PR and release gates independent of baseline-relative trends. -9. Commit versioned machine-readable results and a human-readable report. - -### Out of scope - -TruthBench will not: - -- add an LLM or network call to `fd.clean`, semantic routing, domain repair, - privacy processing, trust scoring, or CI; -- automatically approve or apply an ambiguous repair; -- silently accept documented backend divergence; -- use timing, timestamps, process memory, random salts, or generated identifiers - as decision-determinism inputs; -- replace performance benchmarks with correctness measurements; -- treat an unavailable required backend as a skip; -- overwrite benchmark gold labels to make an implementation pass. - -## Disposition Contract - -Each physical test cell has a stable identifier: -`:::`. - -The four dispositions mean: - -| Disposition | Required default behavior | -|---|---| -| `preserve` | The value is valid, possibly unusual. Mutating its value or semantic meaning is corruption. An error-severity false alarm is also a failure. | -| `repair` | A safe deterministic repair has one gold output. A mutating surface must produce that exact value and dtype; a read-only surface must identify the problem without inventing another value. | -| `flag` | The value must be surfaced with evidence but remain unchanged by default. | -| `review` | The value is ambiguous or policy-conflicted and must enter a human-review, quarantine, reject, or explicit suggestion path without an automatic guess. | - -All pristine background cells default to `preserve`. Adversarial injections replace -that label explicitly. Fixture construction fails if any cell has no label, more -than one label, a missing stable ID, or an invalid expected output. - -Row-level expectations such as exact duplicate removal and schema-level -expectations such as added, missing, renamed, or type-drifted columns are stored as -separate case records. They never substitute for cell labels. - -## Gold Fixture Design - -Each domain fixture contains a pristine frame, an adversarial frame, a complete -cell-label matrix, explicit schemas, context policies, protected columns, PII -canaries, and row/schema expectations. Generation is pure Python with fixed seeds, -fixed reference dates, fixed timezone, fixed locale assumptions, and stable row -identifiers. - -The corpus includes: - -- valid extremes, rare categories, leading-zero identifiers, Unicode names, - uncommon but valid codes, zero-value transactions, and empty free text; -- ambiguous decimal separators, numeric dates, percentages, abbreviated units, - policy aliases, category variants, and case-sensitive identifiers; -- mixed kilograms/pounds, Celsius/Fahrenheit, percentages/fractions, local time - zones, currencies, currency symbols, and Indian/international number grouping; -- English and non-English text, mixed scripts, emoji, combining characters, - mojibake, zero-width characters, HTML entities, and alternate encodings; -- ISO, US, European, textual, partial, timezone-aware, and impossible dates; -- hidden email, phone, SSN/national-ID, card, address, medical, and FERPA-style - identifiers in free text, numeric columns, category values, and late rows; -- added, removed, reordered, renamed, duplicated, and type-drifted columns; -- contradictory protection/repair, currency/unit, range, locale, and mapping - policies; -- semantic traps where `apple`, `Apple`, and `AAPL` have different correct - meanings based on column, schema, and domain. - -Domain emphasis: - -- **Finance:** instruments, company names, tickers, prices, currencies, ledger - signs, percentages, settlement dates, and transaction memos. -- **Healthcare:** patient identifiers, clinical codes, lab values and units, - dates of birth, vital signs, free-text notes, and protected health information. -- **Retail:** SKUs, quantities, prices, promotions, returns, product names, - currencies, reviews, and zero-price edge cases. -- **CRM:** customer IDs, Unicode names, email, phone, country codes, lifecycle - states, signup dates, and free-text contact notes. -- **Logistics:** shipment IDs, UN/LOCODE-like locations, weights, dimensions, - temperatures, time zones, tracking states, delivery windows, and addresses. -- **Government:** case IDs, agency and district codes, postal identifiers, - multilingual values, fiscal amounts, public dates, and restricted identifiers. -- **Education:** student IDs, grades, assessment scales, school years, programs, - enrollment dates, guardian details, and FERPA-protected notes. -- **Insurance:** claim and policy IDs, coverage codes, premiums, reserves, - incident dates, status transitions, medical/loss descriptions, and claimant PII. - -Fixture files may contain synthetic PII canaries because the detector needs raw -inputs. No result, report, failure artifact, or committed baseline may contain -those literals. - -## Public Surface Inventory - -TruthBench maintains a versioned manifest of decision-bearing and decision-sink -surfaces. A contract test compares the manifest with the public exports and known -CLI commands so a new surface cannot be added without an explicit coverage -classification. - -The release profile covers: - -- `fd.clean`, `Cleaner`, `clean_csv`, `CleanResult`, and `CleanReport`; -- fluent pipelines, plan/suggest/apply workflows, and decision hashes; -- field validation and remediation policy; -- validation suites, enterprise contracts, and context policies; -- bundled domain validators and domain repair integration; -- deterministic semantic default, review, and auto behavior; -- safe text cleaning and encoding linting; -- PII detection, anonymization, privacy policies, and privacy reports; -- trust scoring, quality reports, and trust gates; -- streaming with fixed batch partitions; -- pandas, Polars, and DuckDB required backend paths; -- AI Copilot with `provider=None` only; -- generated Copilot code parsing, compilation, and controlled execution; -- report exporters, JSON, Markdown, HTML/Peel rendering, CLI stdout/stderr, and - persisted artifacts as privacy and audit sinks. - -Explicit low-level utilities that perform a caller-requested transform but make no -semantic decision are classified as `explicit-transform`. They receive input/output -and safety contract tests, but they do not pretend to infer a disposition. - -## Surface Adapter Contract - -Every adapter declares: - -- stable name and version; -- whether it mutates, validates, suggests, serializes, or renders; -- supported fixture features and required dependencies; -- the surface-specific mapping from the global gold disposition to the expected - surface decision; -- output extraction, decision extraction, and audit extraction rules; -- deterministic fields and explicitly excluded telemetry fields; -- whether exact backend parity is required. - -For example, a mutating cleaner is expected to repair `repair`, preserve -`preserve`, and leave `flag`/`review` unchanged while surfacing them. A read-only -validator is expected to preserve values and emit `flag` or `review` evidence for -the corresponding defects. A PII detector is expected to flag labelled PII and -remain silent on non-PII preservation traps. - -Adapters must not catch broad exceptions. An unexpected exception is a benchmark -failure with its type, safe message, fixture, surface, and reproduction ID. - -## Normalized Decision Record - -TruthBench emits one record for every `(surface, backend, repeat, cell)` tuple: - -- record schema version; -- run, fixture, case, and cell IDs; -- domain, row ID, and column; -- expected and actual disposition; -- input type and privacy-safe digest; -- expected and actual output type; -- output value for non-sensitive cells or a redacted value plus digest for - sensitive cells; -- confidence, risk, status, model/rule ID, and rationale; -- evidence kinds without raw sensitive samples; -- mutation, detection, quarantine, and human-review booleans; -- audit-required and audit-complete booleans plus matching audit IDs; -- trust before, trust after, and delta at the applicable frame/surface level; -- requested backend, actual backend, fallback events, and backend differences; -- normalized decision hash and repeat-consistency status. - -Exact output equality includes dtype and semantic representation. The comparator -does not equate `"402.10"` with `402.1`, strip whitespace before preservation -checks, or treat case-folded identifiers as identical. Explicit fixture metadata -may authorize a representation equivalence only when that equivalence is itself -the gold repair. - -## Backend Consistency - -The release environment must install and successfully execute pandas, Polars, and -DuckDB. Missing required dependencies, silent materialization, unexpected fallback, -or a mismatched requested/actual backend fails the run. - -Parity uses the common deterministic native subset with -`fallback_policy="error"`. It compares normalized decisions, exact values where -the documented contract promises equality, row identity/order, action status, -confidence, rationale class, and audit coverage. A documented dtype or aggregation -difference is only accepted when represented by an explicit, tested equivalence -rule and disclosed in the report. - -Spark and FreshCore require JVM/native build infrastructure. They belong to the -extended scheduled profile, where absence is a profile failure rather than a skip. -Their adapter contracts and gate-tampering tests remain part of normal pytest. - -## Determinism - -Each deterministic surface runs at least twice in a fresh logical context. -TruthBench compares normalized data, decisions, confidences, rationales, findings, -audit records, ordering, decision hashes, and trust scores. - -Wall-clock duration, peak memory, generated timestamps, run IDs, lineage IDs, and -documented cryptographic randomness are excluded from the decision hash. Privacy -operations that intentionally use random salts must still make the same masking -decision, redact the same spans, disclose the randomness, and produce no raw PII. -Parity tests use an explicit fixed secret so output equality remains testable. - -The default cleaning path, default semantic backend, validators, domain decisions, -trust scores, and generated code must be fully deterministic. - -## Privacy Boundary - -The raw-PII gate scans every sink that could escape the input data boundary: - -- serialized reports and findings; -- action metadata, coerced-cell recovery records, examples, and audit logs; -- Copilot model context, narrative inputs, recommended code, and rendered report; -- plan JSON, validation JSON, domain logs, privacy reports, and quality reports; -- CLI stdout/stderr, Markdown, HTML, JSON, exception tables, and committed results; -- minimized reproductions and failure catalogues. - -The scanner checks exact canaries plus case, whitespace, punctuation, encoding, -Unicode-normalization, and digit-only variants. Sensitive values are represented by -`[REDACTED]` and a one-way run-scoped digest. Explicit in-memory APIs whose purpose -is to return the user's own data remain usable, but their default serialization and -rendering paths must not leak raw PII. Any explicit `include_pii=True`-style escape -hatch is outside the release profile and receives a separate opt-in warning test. - -## Runner Flow - -For each profile, seed, fixture, and surface, the runner: - -1. validates the complete oracle and fixture hash; -2. snapshots the input and protected columns; -3. computes pristine and adversarial trust scores; -4. invokes the surface through its public API; -5. extracts output, decisions, audit evidence, and backend disclosures; -6. verifies input immutability and protected-column byte identity; -7. emits normalized cell records and schema/row case records; -8. reruns deterministic surfaces and compares normalized hashes; -9. compares required backends; -10. scans every sink for PII canaries; -11. evaluates absolute gates; -12. minimizes every failure and writes privacy-safe reproduction artifacts; -13. writes versioned JSON and Markdown results atomically. - -Infrastructure errors fail closed. A partial run cannot report passed gates. - -## Mandatory Release Gates - -The release command fails if any of these conditions is true: - -1. **Valid-value corruption:** any `preserve` cell changes value, dtype, row - identity, or semantic representation, or receives an error-severity false alarm. -2. **Protected-column modification:** any protected cell or protected schema - property differs, even if another oracle labelled the value repairable. -3. **Raw PII leakage:** any canary or normalized variant appears in a scanned sink. -4. **Backend inconsistency:** any required backend produces a divergent decision, - output, ordering, confidence class, rationale class, or audit outcome without an - explicit permitted equivalence. -5. **Default non-determinism:** repeated default decisions or normalized outputs - differ. -6. **Broken generated code:** generated code fails AST parsing, compilation, or - controlled execution against its fixture, performs a network call, or exposes a - PII canary. -7. **Unexplained high confidence:** a decision at confidence `>= 0.90` lacks a - non-empty substantive rationale, rule/model provenance, and matching audit - evidence. Boilerplate or whitespace-only text does not count. -8. **Trust inversion:** an adversarial/corrupted frame scores higher than its - pristine source, or a destructive degenerate output is scored as trustworthy. - -Additional correctness gates require complete cell labels, zero unexpected -exceptions, zero unresolved required backends, 100% audit coverage for mutations, -100% routing of `review` cases, exact repair outputs, zero mutations of `flag` -cases, and internally consistent result counts. - -Absolute gates cannot be waived by a historical baseline. Baselines may detect -additional regressions but cannot make a mandatory failure pass. - -## Failure Reproduction and Minimization - -Every failure receives a stable ID derived from fixture version, surface, backend, -cell/case ID, gate, and normalized evidence. The reducer repeatedly tests the same -public surface while attempting, in order: - -1. removal of unrelated fixtures and policies; -2. removal of unrelated columns; -3. removal of unrelated rows while retaining required role/context evidence; -4. removal of unrelated labelled mutations; -5. simplification of values, schemas, and policy sentences; -6. reduction to one backend and one repeat when those dimensions are not causal. - -The reducer stops at a local one-minimal case or a configured evaluation budget. -It never changes the expected disposition or treats disappearance of the target -cell as success. The minimized artifact includes a sanitized frame, schema, policy, -config, exact command, expected/actual record, likely component, and result hash. - -For every validated implementation failure, the development workflow is: - -1. reproduce through TruthBench; -2. minimize; -3. identify the source-to-decision root cause; -4. add a focused failing pytest regression; -5. implement one root-cause fix; -6. run the focused test, affected suite, TruthBench, and full required tests; -7. update committed TruthBench results and failure catalogue; -8. record the fix, regression test, and remaining limitation. - -Expected outputs are changed only when the written product contract changes and a -human explicitly approves that contract change. They are never changed to conceal -a failure. - -## Result Artifacts - -Committed artifacts live under `benchmarks/truthbench/results/`: - -- `latest.json`: versioned complete machine-readable result; -- `latest.md`: concise metrics, gate status, failures, fixes, and limitations; -- `baseline.json`: fixture/result hashes and regression metrics, not gate waivers; -- `failures/.json`: minimized privacy-safe reproductions for unresolved - or newly fixed failures; -- `README.md`: schema, reproduction, comparison, and update policy. - -The JSON records repository commit, FreshData version, Python and dependency -versions, operating system, profile, seeds, fixture hashes, surface versions, -required backends, gate configuration, command, and normalized decision hashes. -Timestamps are metadata and never part of reproducibility hashes. - -Result verification rejects unknown schema versions, missing fixtures, incomplete -runs, mismatched fixture hashes, inconsistent aggregates, leaked PII, absent gate -results, and claims not backed by cell records. - -## CI and Release Integration - -Normal pytest receives fast contract tests for models, fixture completeness, -adapters, comparators, gates, minimization, serialization, privacy scanning, -generated code, and every fixed implementation failure. - -PR CI runs the deterministic release profile: - -```bash -python -m benchmarks.truthbench run \ - --profile release \ - --backends pandas,polars,duckdb \ - --require-backends \ - --repeats 2 \ - --check -``` - -The production release workflow runs the same command against the exact resolved -release commit before building distributions. A failure prevents publication. - -A scheduled extended profile adds Spark, FreshCore, more seeds, more generated -variants, alternative batch partitions, and larger fixtures. It has no -`continue-on-error`, no silent optional-dependency skip, and publishes its full -result artifacts. - -The Makefile exposes `truthbench`, `truthbench-release`, and -`truthbench-extended` targets. Documentation gives exact local reproduction -commands and distinguishes semantic correctness from performance evidence. - -## Testing Strategy - -TruthBench tests are layered: - -- model invariants and JSON schema round trips; -- complete cell-label and stable-ID validation; -- deterministic fixture and adversarial generator tests; -- exact comparator tests for values, dtypes, Unicode, missing values, dates, and - protected bytes; -- adapter contract tests using small synthetic surface outputs; -- public-surface inventory completeness tests; -- gate-tampering tests proving each mandatory gate fails independently; -- PII scanner mutation tests for every normalized canary variant; -- backend parity and fallback-honesty tests; -- repeated-run determinism tests; -- generated-code parse, compile, controlled-execution, network-denial, and privacy - tests; -- reducer tests proving the target failure remains after minimization; -- end-to-end smoke and full release-profile tests; -- one focused permanent regression for every reproduced FreshData defect. - -Tests must not use broad exception catches, unconditional skips, weakened existing -assertions, or changed gold outputs to make a run green. Optional infrastructure is -selected by profile; within a selected profile it is required. - -## Initial Audit Hypotheses - -Repository inspection identified high-priority hypotheses that TruthBench must -attempt to reproduce before they are called defects: - -- domain repair may occur after the core protected-column verification; -- post-domain report totals may describe the pre-domain frame; -- the 1,000-entry `coerced_cells` cap may also cap quarantine protection and allow - later coerced cells to be imputed; -- semantic review mode may auto-apply high-confidence proposals from non-default - backends; -- Validation Gauntlet semantic actions may not map to cells because semantic - actions aggregate distinct values rather than carrying a row; -- Validation Gauntlet domain findings may be counted without earning cell-level - detection evidence; -- column-level audit attribution may overstate cell-level completeness; -- serialized raw recovery values, text-cleaning originals, validation originals, - plan examples, or privacy groups may cross the intended audit privacy boundary; -- generated timestamps and random masking salts may be confused with decision - non-determinism unless normalization is explicit; -- plan signatures that sample only early rows may miss tail-only drift. - -Each hypothesis is subject to the reproduce/minimize/root-cause workflow. No source -change is justified solely by inspection. - -## Acceptance Criteria - -TruthBench is complete when: - -- all eight domain fixtures exist and every cell has one valid disposition; -- the public-surface manifest has no unclassified decision-bearing surface; -- every required comparison dimension appears in each applicable decision record; -- pandas, Polars, and DuckDB complete the release profile with no inconsistent - decisions or undisclosed fallback; -- all eight mandatory gates have independent negative tests and pass the real run; -- all discovered implementation failures have minimized reproductions, root-cause - fixes, focused pytest regressions, and updated benchmark results; -- generated code compiles and executes in the controlled offline harness; -- committed artifacts contain no planted PII; -- default behavior is deterministic under normalized repeat comparison; -- trust never increases after labelled corruption; -- the existing test suite, Validation Gauntlet, CleanBench, and benchmark tests are - not weakened; -- PR CI and the production release workflow execute TruthBench as a required gate; -- final documentation lists implemented files, failures, fixes, commands, results, - limitations, and evidence for every release gate. diff --git a/freshdata-benchmarks/.github/workflows/benchmark.yml b/freshdata-benchmarks/.github/workflows/benchmark.yml deleted file mode 100644 index b52c5bc..0000000 --- a/freshdata-benchmarks/.github/workflows/benchmark.yml +++ /dev/null @@ -1,186 +0,0 @@ -name: Benchmarks -on: - push: - branches: [main] - pull_request: - branches: [main] - schedule: - - cron: '0 6 * * 1' # Weekly Monday 6am UTC - -permissions: - contents: write - pages: write - id-token: write - -concurrency: - group: benchmark-${{ github.ref }} - cancel-in-progress: true - -jobs: - # ───────────────────────────────────────────── - # Quick benchmark run on every PR - # ───────────────────────────────────────────── - benchmark-quick: - if: github.event_name == 'pull_request' - runs-on: ubuntu-latest - timeout-minutes: 30 - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - - uses: actions/setup-python@v5 - with: - python-version: '3.12' - cache: pip - - - name: Install dependencies - run: | - pip install --upgrade pip - pip install -r requirements.txt - pip install asv virtualenv - - - name: Configure ASV machine - run: asv machine --yes - - - name: Run quick benchmarks - run: asv run --quick --bench "BenchmarkPipeline" -e 2>&1 | tee benchmark_output.txt - - - name: Post benchmark results as PR comment - if: always() - uses: actions/github-script@v7 - with: - script: | - const fs = require('fs'); - let output = ''; - try { - output = fs.readFileSync('benchmark_output.txt', 'utf8'); - } catch (e) { - output = 'Benchmark output not available.'; - } - // Truncate if too long for a comment - if (output.length > 60000) { - output = output.substring(0, 60000) + '\n... (truncated)'; - } - github.rest.issues.createComment({ - issue_number: context.issue.number, - owner: context.repo.owner, - repo: context.repo.repo, - body: `## ⚡ Benchmark Results (Quick)\n\n\`\`\`\n${output}\n\`\`\`` - }); - - # ───────────────────────────────────────────── - # Focused freshdata-vs-pandas comparison - # ───────────────────────────────────────────── - benchmark-focused: - if: github.event_name == 'push' || github.event_name == 'schedule' - runs-on: ubuntu-latest - timeout-minutes: 60 - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - - uses: actions/setup-python@v5 - with: - python-version: '3.12' - cache: pip - - - name: Run focused benchmarks - run: bash bench/run_benchmarks.sh - - - name: Upload focused benchmark artifacts - uses: actions/upload-artifact@v4 - with: - name: benchmark-focused-${{ github.sha }} - path: | - bench/artifacts/ - .asv/html/ - reports/ - retention-days: 90 - - # ───────────────────────────────────────────── - # Full benchmark run on push to main + weekly - # ───────────────────────────────────────────── - benchmark-full: - if: github.event_name == 'push' || github.event_name == 'schedule' - runs-on: ubuntu-latest - timeout-minutes: 120 - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - - uses: actions/setup-python@v5 - with: - python-version: '3.12' - cache: pip - - - name: Install dependencies - run: | - pip install --upgrade pip - pip install -r requirements.txt - pip install asv virtualenv - - - name: Configure ASV machine - run: asv machine --yes - - - name: Run full benchmarks - run: | - asv run --skip-existing-successful -e 2>&1 | tee benchmark_full_output.txt - - - name: Generate reports - run: | - python scripts/generate_reports.py - asv publish - - - name: Upload benchmark results - uses: actions/upload-artifact@v4 - with: - name: benchmark-results-${{ github.sha }} - path: | - .asv/results/ - reports/ - retention-days: 90 - - - name: Deploy HTML report to GitHub Pages - if: github.ref == 'refs/heads/main' - uses: peaceiris/actions-gh-pages@v4 - with: - github_token: ${{ secrets.GITHUB_TOKEN }} - publish_dir: .asv/html - force_orphan: true - commit_message: "Update benchmark report (${{ github.sha }})" - - # ───────────────────────────────────────────── - # Generate plots (separate job to avoid timeout) - # ───────────────────────────────────────────── - generate-plots: - needs: benchmark-full - if: github.ref == 'refs/heads/main' - runs-on: ubuntu-latest - timeout-minutes: 15 - steps: - - uses: actions/checkout@v4 - - - uses: actions/setup-python@v5 - with: - python-version: '3.12' - - - name: Install plot dependencies - run: pip install matplotlib numpy tabulate - - - name: Download benchmark results - uses: actions/download-artifact@v4 - with: - name: benchmark-results-${{ github.sha }} - - - name: Generate plots - run: python scripts/generate_plots.py - - - name: Upload plots - uses: actions/upload-artifact@v4 - with: - name: benchmark-plots-${{ github.sha }} - path: reports/plots/ - retention-days: 90 diff --git a/freshdata-benchmarks/README.md b/freshdata-benchmarks/README.md index 08053a0..cb16d09 100644 --- a/freshdata-benchmarks/README.md +++ b/freshdata-benchmarks/README.md @@ -1,9 +1,10 @@ # FreshData Benchmark Suite -[![Benchmarks](https://img.shields.io/badge/📊_ASV_Benchmarks-results-blue?style=for-the-badge)](https://FreshCode-Org.github.io/freshdata/) -[![CI](https://img.shields.io/github/actions/workflow/status/FreshCode-Org/freshdata/benchmark.yml?label=Benchmark%20CI&style=flat-square)](https://github.com/FreshCode-Org/freshdata/actions/workflows/benchmark.yml) - -A comprehensive, scientifically rigorous comparative benchmark suite for `freshdata-cleaner` using [Airspeed Velocity (ASV)](https://asv.readthedocs.io/). +A comparative benchmark suite for `freshdata-cleaner` using +[Airspeed Velocity (ASV)](https://asv.readthedocs.io/). This suite is run +locally and its raw ASV results are committed under `.asv/results/`; it is +separate from the repository's CI `Benchmark` workflow, which runs the +[CleanBench harness](../benchmarks/) in `benchmarks/`. This suite measures the performance of FreshData's data cleaning operations against the Python data ecosystem: - **Pandas** (baseline) @@ -13,12 +14,6 @@ This suite measures the performance of FreshData's data cleaning operations agai - **Feature-engine** - **AutoClean** -## 📊 README Badge (copy-paste) - -```markdown -[![Benchmarks](https://img.shields.io/badge/📊_ASV_Benchmarks-results-blue?style=for-the-badge)](https://FreshCode-Org.github.io/freshdata/) -``` - ## Quick Start ```bash @@ -64,7 +59,7 @@ The headline benchmark compares **FreshData** and **Pandas** head-to-head across # One-shot: installs deps, runs benchmarks, generates reports bash bench/run_benchmarks.sh -# Quick mode (fewer iterations, for CI validation) +# Quick mode (fewer iterations, for fast validation) bash bench/run_benchmarks.sh --quick ``` @@ -75,7 +70,7 @@ To reproduce benchmark results exactly: ```bash # Clone and enter the repo git clone https://github.com/FreshCode-Org/freshdata.git -cd freshdata-benchmarks +cd freshdata/freshdata-benchmarks # Pin all randomness and threading for deterministic results export PYTHONHASHSEED=42 @@ -137,15 +132,13 @@ The suite covers 15 domains using synthetic datasets (10K to 10M rows) with real 14. **Group Aggregations** (`benchmark_groupagg.py`) - Groupby/profile comparisons 15. **FreshData vs Pandas** (`benchmark_freshdata_vs_pandas.py`) - Focused 6-operation head-to-head -## CI/CD and Published Results - -Benchmarks are executed weekly via GitHub Actions on standard CI hardware (2 vCPU, 7GB RAM). -Results are automatically published to the `gh-pages` branch. +## Published Results -Three CI jobs run the benchmarks: -- **benchmark-quick** — PR validation (pipeline only, fast feedback) -- **benchmark-focused** — FreshData-vs-Pandas head-to-head comparison -- **benchmark-full** — Complete multi-library suite +This ASV suite is not wired into CI: run it locally with the commands above +and browse the results with `asv publish && asv preview`. Raw per-machine +results live in `.asv/results/` so runs are comparable over time. The +repository's CI `Benchmark` workflow runs the separate CleanBench harness +(`benchmarks/` at the repo root) on every push to `main` and weekly. ## Methodology diff --git a/pyproject.toml b/pyproject.toml index 937c350..e6ceb8b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -233,19 +233,21 @@ artifacts = [ ] [tool.hatch.build.targets.sdist] +# Patterns are anchored with a leading "/": hatchling matches unanchored +# patterns at any depth (a bare "examples" would also ship docs/examples). include = [ - "src", - "crates/freshcore", - "tests", - "examples", - "README.md", - "CHANGELOG.md", - "LICENSE", - "SECURITY.md", - "RELEASE.md", - "CONTRIBUTING.md", - "CONTRIBUTING_DOMAINS.md", - "CODE_OF_CONDUCT.md", + "/src", + "/crates/freshcore", + "/tests", + "/examples", + "/README.md", + "/CHANGELOG.md", + "/LICENSE", + "/SECURITY.md", + "/RELEASE.md", + "/CONTRIBUTING.md", + "/CONTRIBUTING_DOMAINS.md", + "/CODE_OF_CONDUCT.md", ] [tool.pytest.ini_options]