Validation Gauntlet: gold-labelled disposition benchmark + six defect fixes

[JohnnyWilson16]

Summary

A release-ready Validation Gauntlet: an objective, gold-labelled benchmark for FreshData's semantic validation, domain validation and text-cleaning surfaces — plus fixes for the six highest-impact defects it exposed, each pinned by a failing-first regression test.

The benchmark (benchmarks/gauntlet/)

Five deterministic fixtures (finance, healthcare, crm, ecommerce, adversarial text; 300 rows, seeded) label every injected defect with the disposition FreshData should choose: preserve / repair / flag / review. Automatic removal is never the default correct answer — mutating a flag/review/preserve cell counts as a corruption, the most severe verdict.

Flagship case: apple in a price column is quarantined (original kept in report.coerced_cells), Apple (company) and AAPL (ticker) are preserved, lowercase ticker apple is routed to review with suggestion AAPL.

The runner drives public API only: fd.clean defaults, fd.validate_fields, fd.clean_text (safe + explicit opt-in config, graded separately), semantic auto mode, lint_text_encoding, detect_pii, domain packs. Metrics: detection P/R/F1, repair accuracy (per surface), review routing, preservation rate, corruption count, escape rate, FPR, audit completeness, determinism, trust monotonicity, runtime + peak memory. python -m benchmarks.gauntlet run emits JSON + Markdown with a failure catalogue; --check gates against absolute thresholds and the committed baseline.json. gauntlet.yml runs it on every PR with a job summary and artifacts.

Defects fixed (failing regression test first — tests/test_gauntlet_regressions.py)

1. Unparseable values were laundered into fabricated statistics (P1). fix_dtypes nulled coercion casualties and the auto engine imputed the median: 'apple' price → 425.17, 'forty' age → 53, '37,2' °C → 37.6. Now quarantined: left missing, excluded from auto-imputation, originals in the new CleanReport.coerced_cells, human_review audit action. Sentinels/true-NaN keep documented auto-impute; explicit impute= still fills everything.
2. Formatted-number stragglers were destroyed instead of parsed: the existing "$1,234.56" rescue only ran when the plain parse failed its threshold; '$3,400.50' in a mostly-plain column became NaN. Now rescued into correct numbers.
3. The documented validate_fields handoff found nothing: fd.clean's contamination warning fires from a 60% parse share, but the consensus gate needed 80% — the README frame the warning names passed silently. Consensus now honours the same straggler boundary.
4. Explicit vocabulary lost to null-marker heuristics: 'NA' with allowed_values={'US','DE','NA'} was treated as missing. Membership now outranks null markers (Namibia wins).
5. Case variants of allowed values were silently accepted: 'ACTIVE' vs {'active'} now yields a warning-severity issue with the canonical suggestion (accept_with_warning).
6. No date bounds: FieldSpec.min_value/max_value accept date strings for date fields; a future dob or an 1875 admission date is now flagged.
7. Typography was rewritten in content fields: the punctuation→ASCII map turned 12″ into 12" and em-dashes into hyphens inside free text and entity names. Content-bearing types now keep their typography; untyped columns keep existing behaviour.

Results (300 rows/fixture, seed 42)

fixture	P	R	F1	repair	review	preserve	corrupt	escape	audit	det	trust mono
crm	1.0	0.917	0.957	1.0	1.0	1.0	0	0.083	1.0	✅	✅
ecommerce	1.0	1.0	1.0	1.0	1.0	1.0	0	0	1.0	✅	✅
finance	1.0	1.0	1.0	1.0	1.0	1.0	0	0	1.0	✅	✅
healthcare	1.0	1.0	1.0	1.0	1.0	1.0	0	0	1.0	✅	✅
text	1.0	1.0	1.0	1.0	—	1.0	0	0	1.0	✅	✅

Before the fixes the same gauntlet measured 11+ corruptions across fixtures (median-fabricated prices, 'NA'→'Unknown' countries, mangled product names) and repair accuracy as low as 0.14 on finance.

Known accepted gap (documented in the failure catalogue and docs): a hostile SQL-injection payload inside a person_name field escapes detection — any plausibility check tight enough to catch it false-positives on legally real names (X Æ A-12).

Verification

• pytest -m "not online and not large": 3490 passed, 0 failed, coverage 93.68% (gate 93) — baseline before changes: 3463 passed
• ruff check src tests benchmarks/gauntlet: clean
• mypy src/freshdata: clean (199 files)
• python -m benchmarks.gauntlet run --check: all gates pass
• Existing benchmarks/bench.py run fixtures: 100% fidelity / 0% false repair / 100% preservation (unchanged)

Backward compatibility

• Cells that previously came back imputed after failing dtype parsing now come back missing with a review action + recoverable originals — a deliberate, changelog-documented behaviour change; impute="median" (explicit) restores the old outcome.
• clean_text(df, field_types={...: "free_text"}) no longer normalizes typographic punctuation for content types; untyped columns unchanged.
• report.to_dict() gains a coerced_cells key only when non-empty.

Built with AI tooling; all changes reviewed and verified by test runs above.

🤖 Generated with Claude Code

[strix-security]

Strix is installed on this repository, but we couldn't run this PR security review because this workspace's trial has ended. Add a card to resume code reviews here.

[coderabbitai]

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 25228349-d75f-4def-baf6-dcec59c87381

📥 Commits

Reviewing files that changed from the base of the PR and between 30d82b5 and 842a1d3.

📒 Files selected for processing (20)

• .github/workflows/gauntlet.yml
• .gitignore
• CHANGELOG.md
• benchmarks/README.md
• benchmarks/gauntlet/__init__.py
• benchmarks/gauntlet/__main__.py
• benchmarks/gauntlet/baseline.json
• benchmarks/gauntlet/fixtures.py
• benchmarks/gauntlet/metrics.py
• benchmarks/gauntlet/report.py
• benchmarks/gauntlet/runner.py
• docs/validation-gauntlet.md
• mkdocs.yml
• src/freshdata/engine/missing.py
• src/freshdata/fieldcheck.py
• src/freshdata/report.py
• src/freshdata/steps/dtypes.py
• src/freshdata/textclean.py
• tests/benchmark/test_gauntlet.py
• tests/test_gauntlet_regressions.py

 _________________________________________________
< PhD, MSc, BSc, and a black belt in code review. >
 -------------------------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       / 　 づ

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/validation-gauntlet-jwd

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — 2026-07-13T04:16:44Z

• freshdata: 1.1.1
• python: 3.12.13
• platform: Linux-6.17.0-1018-azure-x86_64-with-glibc2.39

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %
crm	10,200	40	1.406	1.411	13.9	100.0	0.0	100.0	93.84	✅	100.0
event_log	10,000	25	0.568	0.569	5.1	100.0	0.0	100.0	99.677	✅	100.0
finance	10,200	60	1.809	1.815	18.3	100.0	0.0	100.0	99.499	✅	100.0
gold	10,200	7	0.214	0.215	3.7	100.0	0.0	100.0	98.3	✅	100.0
provenance	10,000	18	0.467	0.471	7.9	100.0	0.0	100.0	99.765	✅	100.0
wide_schema	10,000	100	3.446	3.474	13.7	100.0	0.0	100.0	96.042	✅	100.0

Authored-code reduction (Metric 6)

• FreshData: 3 lines
• pandas baseline: 25 lines (88.0% reduction)