feat: add TruthBench semantic red-team foundation

[Rohith27216]

Summary

• Add TruthBench oracle models, schema validation, and deterministic eight-domain adversarial fixtures.
• Add privacy scanning/redaction, public-surface inventory, adapter protocol, cleaning/validation/privacy/reporting/Copilot adapters, and strict pandas/Polars/DuckDB parity.
• Add fail-closed normalization and release-gate evaluation with regression coverage for partial runs, PII leaks, trust inversion, backend divergence, unaudited mutations, and unexplained high-confidence decisions.
• Isolate duplicate test module names so the repository-wide pytest collection is stable.

Verification

• PYTHONPATH=src python -m pytest -m "not online and not large": passed; the two previously recorded timing flakes did not reproduce in this run.
• PYTHONPATH=src python -m pytest -q tests/truthbench --no-cov: passed.
• ruff check .: passed.
• mypy src/freshdata: passed.
• python -m build --no-isolation: passed.

Scope note

This draft contains the implemented/reviewed foundation through Tasks 1–11. Determinism/generated-code execution, the full runner/minimizer/CLI, artifact publication, CI release wiring, source-level audit fixes, and final release-profile verification remain follow-up work.

Summary by CodeRabbit

• New Features

  • Added a deterministic TruthBench benchmark covering eight business domains and common data-quality scenarios.
  • Added privacy-safe scanning and redaction to detect sensitive-value leaks without exposing the underlying data.
  • Added standardized surface adapters, backend parity checks, normalized results, and comprehensive release-gate validation.
  • Added strict schema, integrity, reproducibility, and audit checks for benchmark results.

• Documentation

  • Added design, implementation-plan, and completion reports documenting benchmark coverage and verification.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 414a27c2-bd0c-4ce8-8b1f-fbcd50c3dca3

📥 Commits

Reviewing files that changed from the base of the PR and between a1da862 and 0eae829.

📒 Files selected for processing (43)

• .superpowers/sdd/task-4-report.md
• .superpowers/sdd/task-5-report.md
• .superpowers/sdd/task-6-report.md
• benchmarks/truthbench/__init__.py
• benchmarks/truthbench/exact.py
• benchmarks/truthbench/fixtures/__init__.py
• benchmarks/truthbench/fixtures/base.py
• benchmarks/truthbench/fixtures/crm.py
• benchmarks/truthbench/fixtures/education.py
• benchmarks/truthbench/fixtures/finance.py
• benchmarks/truthbench/fixtures/government.py
• benchmarks/truthbench/fixtures/healthcare.py
• benchmarks/truthbench/fixtures/insurance.py
• benchmarks/truthbench/fixtures/logistics.py
• benchmarks/truthbench/fixtures/retail.py
• benchmarks/truthbench/gates.py
• benchmarks/truthbench/inventory.py
• benchmarks/truthbench/models.py
• benchmarks/truthbench/normalize.py
• benchmarks/truthbench/privacy.py
• benchmarks/truthbench/schema.py
• benchmarks/truthbench/surfaces/__init__.py
• benchmarks/truthbench/surfaces/backends.py
• benchmarks/truthbench/surfaces/base.py
• benchmarks/truthbench/surfaces/cleaning.py
• benchmarks/truthbench/surfaces/copilot.py
• benchmarks/truthbench/surfaces/privacy.py
• benchmarks/truthbench/surfaces/reporting.py
• benchmarks/truthbench/surfaces/validation.py
• docs/superpowers/plans/2026-07-14-freshdata-truthbench.md
• docs/superpowers/specs/2026-07-14-freshdata-truthbench-design.md
• tests/benchmark/__init__.py
• tests/truthbench/__init__.py
• tests/truthbench/conftest.py
• tests/truthbench/test_backends_determinism.py
• tests/truthbench/test_fixtures.py
• tests/truthbench/test_gates.py
• tests/truthbench/test_inventory.py
• tests/truthbench/test_models_exact.py
• tests/truthbench/test_normalize.py
• tests/truthbench/test_privacy.py
• tests/truthbench/test_schema.py
• tests/truthbench/test_surface_adapters.py

---

📝 Walkthrough

Walkthrough

TruthBench adds deterministic typed models, eight-domain fixtures, privacy-safe sink scanning, public-surface adapters, backend parity checks, normalization, schemas, release gates, tests, and design documentation.

Changes

TruthBench evaluation framework

Layer / File(s)	Summary
Core contracts and deterministic fixtures benchmarks/truthbench/models.py, benchmarks/truthbench/exact.py, benchmarks/truthbench/schema.py, benchmarks/truthbench/fixtures/*	Adds typed serialization, exact equality, fixture validation/building, schema integrity checks, and deterministic gold fixtures for eight domains.
Privacy-safe sink scanning benchmarks/truthbench/privacy.py, tests/truthbench/test_privacy.py	Scans normalized canary variants across nested sinks and pandas structures, and redacts matches with keyed digest markers.
Surface execution and backend parity benchmarks/truthbench/inventory.py, benchmarks/truthbench/surfaces/*, tests/truthbench/test_surface_adapters.py, tests/truthbench/test_backends_determinism.py	Discovers public surfaces, registers adapters, captures sanitized observations, runs provider-free Copilot analysis, and compares backend execution evidence.
Normalization and release gates benchmarks/truthbench/normalize.py, benchmarks/truthbench/gates.py, tests/truthbench/test_normalize.py, tests/truthbench/test_gates.py	Converts observations into decision/case records and evaluates completeness, integrity, privacy, mutation, determinism, audit, trust, and parity gates.
Validation and documentation tests/truthbench/*, docs/superpowers/*, .superpowers/sdd/*	Adds contract coverage for models, fixtures, schemas, adapters, gates, and privacy behavior, alongside TruthBench design, implementation, and task reports.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant FixtureBuilder
  participant SurfaceAdapter
  participant SinkScanner
  participant Normalizer
  participant ReleaseGates

  FixtureBuilder->>SurfaceAdapter: provide validated fixture
  SurfaceAdapter->>SinkScanner: scan and redact observation sinks
  SurfaceAdapter-->>Normalizer: return SurfaceObservation
  Normalizer-->>ReleaseGates: produce normalized records and cases
  ReleaseGates-->>ReleaseGates: evaluate independent release gates

Loading

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/truthbench-jwd

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)

[strix-security]

Strix is installed on this repository, but we couldn't run this PR security review because this workspace's trial has ended. Add a card to resume code reviews here.