Text-cleaning pipeline + context-aware per-cell field validation

[JohnnyWilson16]

Summary

Adds a production-grade text data cleaning layer and context-aware per-cell field validation, plus a root-cause fix in the core engine that stops a contaminated numeric column from passing through silently.

The driving scenario: a scraped feed inserts "apple" into different columns of a financial pipeline. The verdict must depend on the field —

field	value	verdict
transaction_amount	"apple"	semantic_mismatch → quarantine (never coerced)
company_name	"Apple" / "apple"	valid — real-word names are never rejected for being words
stock_ticker	"AAPL"	valid; optional injected reference universe
stock_ticker	"apple"	domain_mismatch; suggests AAPL only from a caller-supplied trusted mapping
transaction_description	"Payment to Apple"	valid free text; spacing normalized, entity text preserved
numeric column + one "apple"	—	flagged with exact row/column even without a schema (column consensus)

What's in here (4 commits)

1. fix_dtypes silence fix (steps/dtypes.py): a column that is ≥60% numeric but blocked from conversion by a few unparseable cells (≤ max(3, 10%)) now emits a report warning naming the values and rows. Mixed alphanumeric ID columns (e.g. titanic ticket) stay silent. Nothing is coerced.
2. fd.clean_text / fd.clean_text_value / TextCleanConfig (textclean.py): deterministic ordered pipeline — Unicode NFC, control/zero-width/bidi stripping, punctuation normalization, optional HTML/URL removal, casing, repeat capping, truncation, custom ops. Input never mutated; every changed cell logged with original + cleaned + ordered transforms. Field-aware: lossy ops are automatically withheld from structural types (amounts, identifiers, emails, URLs, dates, tickers) and entity names. Stdlib + pandas only.
3. fd.validate_fields / FieldSpec / RemediationPolicy / fd.apply_field_policy (fieldcheck.py): per-cell validation combining declared specs (semantic type, range, vocabulary, pattern, per-field null markers, injected reference/suggest) with value shape and column consensus. Seven distinct failure classes (no generic "outlier" bucket); rare/extreme values are warned, never auto-rejected; configurable non-destructive actions (accept … manual_review); apply_field_policy splits accepted/quarantined/rejected/needs-review copies with a full audit trail. Vectorized per-column pre-screen keeps clean feeds at bulk speed; the per-cell explanation path runs only on suspects. Issues export as QualityFinding (step fieldcheck) into the existing quality-ops exporters.
4. Docs + benchmark: docs/field-validation.md (examples executed and verified against the API) and benchmarks/bench_fieldcheck.py, a seeded labeled-corruption benchmark.

Benchmark (seeded, reproducible: python benchmarks/bench_fieldcheck.py 100000)

validator	precision	recall	F1	rows/s (100k)
regex per column	1.00	0.74	0.85	~499k
pandas coercion	1.00	0.55	0.71	~1.3M
fd.validate_fields	1.00	1.00	1.00	~118k

Baselines miss out-of-range rates, unknown currencies and impossible-but-well-shaped dates; validate_fields catches all labeled corruptions at ~4× the cost of bare regex (after the vectorized pre-screen; it was ~130× before). Peak extra RSS at 100k rows: ~20 MB on a ~33 MB frame.

Tests

93 new tests: the six "apple" cases, a nine-domain dirty-data matrix (finance, insurance, healthcare, e-commerce, support, HR, telecom, logistics, general business), adversarial inputs (homoglyphs, SQL/script/prompt payloads, bidi overrides, 100k-char strings), boundary cases (leap days, empty/one-row/all-null/constant/mixed columns), policy semantics (null-replacement audited, reject/quarantine/review splits), a sqlite ingestion simulation (batch, duplicates, schema change, new category, 5k-row batch), and a regression test proving the old silent-contamination behavior is fixed.

ruff clean; mypy clean on the new/changed modules; wheel builds; full suite green locally (coverage gate short of 93% locally only because optional-dep tests skip, as usual).

Compatibility

Purely additive public API; no existing behavior changes except the new (report-only) warning in fix_dtypes. One golden snapshot was intentionally not regenerated — the warning guard was tightened until titanic's ticket column stayed silent.

Summary by CodeRabbit

• New Features
  • Added configurable, field-aware text cleaning with Unicode, whitespace, HTML, URL, punctuation, and case handling.
  • Added per-cell field validation with schema checks, semantic and domain validation, anomaly detection, and cross-field rules.
  • Added configurable remediation workflows for accepted, quarantined, rejected, and review-needed records, with audit trails.
  • Added detailed validation and cleaning reports for summaries, findings, and tabular export.
• Bug Fixes
  • Added warnings when mostly numeric columns remain text due to unparseable values.
• Documentation
  • Added a guide covering text cleaning, field validation, remediation, reporting, and known limitations.

[strix-security]

Strix Security Review

1 open security finding on this PR:

• 🔵 MEDIUM Regular Expression Denial of Service in field pattern validation — src/freshdata/fieldcheck.py:400

Review summary

The PR adds new text-cleaning and per-cell field-validation paths, plus a warning path in dtype inference. I confirmed one issue in Unsafe regular expression execution in FieldSpec.pattern validation: caller-supplied regex patterns are executed directly against untrusted cell values in both the vectorized pre-screen and the per-cell validation path, which can let a catastrophic pattern stall the validation pipeline.

_{Updated for badbb21.}

---

Reviewed by Strix
_{Re-run review · Configure security review settings}

[coderabbitai]

Warning

Review limit reached

@JohnnyWilson16, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 33 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e8bd2b48-ceec-481a-9ab7-a5d010c9c7f6

📥 Commits

Reviewing files that changed from the base of the PR and between badbb21 and 5af1b4f.

📒 Files selected for processing (2)

• src/freshdata/fieldcheck.py
• tests/test_fieldcheck.py

📝 Walkthrough

Walkthrough

Adds field-aware text cleaning, schema-based validation, remediation policies, audit/reporting outputs, numeric contamination warnings, public exports, documentation, tests, and a reproducible validation benchmark.

Changes

Data quality pipeline

Layer / File(s)	Summary
Field-aware text cleaning src/freshdata/textclean.py, tests/test_textclean.py	Adds configurable, audited scalar and DataFrame text cleaning with field-type safeguards, reports, copy semantics, and adversarial-input coverage.
Validation contracts and cell checks src/freshdata/fieldcheck.py, tests/test_fieldcheck.py	Adds FieldSpec, validation policies, issue/report types, type detection, per-cell checks, consensus inference, and schema validation.
Validation aggregation and remediation src/freshdata/fieldcheck.py, tests/test_fieldcheck.py	Adds rare/outlier and cross-field checks, row-level action aggregation, policy result partitions, audit records, reporting exports, and ingestion-flow tests.
Cleaning warnings and public API src/freshdata/steps/dtypes.py, src/freshdata/__init__.py	Adds numeric-contamination warnings and re-exports the new cleaning and validation APIs.
Documentation and validation benchmark docs/field-validation.md, mkdocs.yml, benchmarks/bench_fieldcheck.py	Documents the APIs and limitations, adds guide navigation, and provides seeded accuracy and throughput comparisons.
Estimated code review effort: 4 (Complex)	~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant IngestionPipeline
  participant validate_fields
  participant FieldValidationReport
  participant apply_field_policy
  participant PolicyResult
  IngestionPipeline->>validate_fields: validate DataFrame with schema and policy
  validate_fields-->>FieldValidationReport: classified cell issues and audits
  IngestionPipeline->>apply_field_policy: apply report to original DataFrame
  apply_field_policy->>FieldValidationReport: aggregate row actions
  apply_field_policy-->>PolicyResult: partition rows and retain audit records

Loading

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 18.80% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	The description is detailed, but it omits the required Type of Change and Checklist sections from the template.	Add the missing Type of Change and Checklist sections, and include a Fixes # reference if an issue number exists.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and accurately reflects the main additions: text cleaning and context-aware field validation.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/textclean-context-validation-jwd

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[strix-security]

Strix flagged 2 new security findings below. See the pinned summary comment for the full PR status.

[github-actions]

FreshData benchmark report — 2026-07-11T18:52:27Z

• freshdata: 1.1.1
• python: 3.12.13
• platform: Linux-6.17.0-1018-azure-x86_64-with-glibc2.39

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %
crm	10,200	40	1.480	1.482	13.8	100.0	0.0	100.0	93.84	✅	100.0
event_log	10,000	25	0.616	0.637	5.1	100.0	0.0	100.0	99.677	✅	100.0
finance	10,200	60	1.930	1.938	18.3	100.0	0.0	100.0	99.499	✅	100.0
gold	10,200	7	0.224	0.225	3.7	100.0	0.0	100.0	98.3	✅	100.0
provenance	10,000	18	0.492	0.492	7.2	100.0	0.0	100.0	99.765	✅	100.0
wide_schema	10,000	100	3.663	3.688	13.7	100.0	0.0	100.0	96.042	✅	100.0

Authored-code reduction (Metric 6)

• FreshData: 3 lines
• pandas baseline: 25 lines (88.0% reduction)

[coderabbitai]

🧹 Nitpick comments (2)

tests/test_fieldcheck.py (1)

305-314: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Remove unused good dict in test_clean_frames_produce_zero_issues

The good variable is created on line 309 and deleted on line 314 without ever being read. This appears to be leftover scaffolding.

♻️ Proposed cleanup

 def test_clean_frames_produce_zero_issues():
     for domain, (df, schema, _) in DOMAIN_CASES.items():
         clean_rows = df.drop(index=df.index[-1])
-        good = {
-            "finance": None, "healthcare": None,  # earlier rows fully clean
-        }
         report = validate_fields(clean_rows.iloc[:1], schema)
         errors = [i for i in report.issues if i.severity == "error"]
         assert errors == [], f"{domain}: clean data flagged: {errors}"
-        del good

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_fieldcheck.py` around lines 305 - 314, Remove the unused good
dictionary and its corresponding del statement from
test_clean_frames_produce_zero_issues, leaving the clean_rows, validate_fields,
and error assertions unchanged.

src/freshdata/textclean.py (1)

210-211: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

normalize_punctuation step also normalizes irregular whitespace

The transform _IRREGULAR_WS_RE.sub(" ", out.translate(_PUNCT_MAP)) does two things: Unicode→ASCII punctuation mapping and irregular whitespace replacement. When only whitespace changes (e.g., NBSP→space), the audit trail records "normalize_punctuation", which is misleading. Test line 33 confirms this: "a b" (with U+00A0) gets transform ("normalize_punctuation",) with no punctuation involved.

Consider splitting into separate steps or renaming to reflect both operations.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/freshdata/textclean.py` around lines 210 - 211, Update the
normalize_punctuation flow to separate punctuation mapping from
irregular-whitespace replacement, so whitespace-only changes are recorded under
an accurate transform name rather than normalize_punctuation. Preserve the
existing punctuation normalization behavior and audit both operations
independently using the surrounding step mechanism.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/freshdata/textclean.py`:
- Around line 210-211: Update the normalize_punctuation flow to separate
punctuation mapping from irregular-whitespace replacement, so whitespace-only
changes are recorded under an accurate transform name rather than
normalize_punctuation. Preserve the existing punctuation normalization behavior
and audit both operations independently using the surrounding step mechanism.

In `@tests/test_fieldcheck.py`:
- Around line 305-314: Remove the unused good dictionary and its corresponding
del statement from test_clean_frames_produce_zero_issues, leaving the
clean_rows, validate_fields, and error assertions unchanged.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: b75b7306-0dce-428f-ba59-aa8a9d8ff60e

📥 Commits

Reviewing files that changed from the base of the PR and between f14d120 and badbb21.

📒 Files selected for processing (9)

• benchmarks/bench_fieldcheck.py
• docs/field-validation.md
• mkdocs.yml
• src/freshdata/__init__.py
• src/freshdata/fieldcheck.py
• src/freshdata/steps/dtypes.py
• src/freshdata/textclean.py
• tests/test_fieldcheck.py
• tests/test_textclean.py