diff --git a/.superpowers/sdd/task-4-report.md b/.superpowers/sdd/task-4-report.md new file mode 100644 index 0000000..7418575 --- /dev/null +++ b/.superpowers/sdd/task-4-report.md @@ -0,0 +1,110 @@ +# Task 4 Report: Add finance, healthcare, retail, and CRM gold datasets + +## Status + +DONE. The four deterministic 16-row domain builders are registered and covered by focused and adjacent TruthBench tests. + +## TDD evidence + +### RED + +```text +PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov +``` + +Observed 16 failures before implementation: each new domain was absent from the fixture registry (`FixtureError: unknown fixture domain`). + +### GREEN + +```text +PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov +``` + +Result: `28 passed`. + +```text +PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py tests/truthbench/test_models_exact.py tests/truthbench/test_schema.py -q --no-cov +``` + +Result: all focused, model, and schema tests green. + +Static checks: + +```text +python -m ruff check benchmarks/truthbench/fixtures tests/truthbench/test_fixtures.py +python -m ruff format --check benchmarks/truthbench/fixtures tests/truthbench/test_fixtures.py +mypy --ignore-missing-imports --explicit-package-bases benchmarks/truthbench/fixtures tests/truthbench/conftest.py tests/truthbench/test_fixtures.py +git diff --check +``` + +All passed. + +## Files + +- `benchmarks/truthbench/fixtures/finance.py` — finance frame and oracle labels for Apple semantic traps, numeric/currency/date defects, protected ticker conflict, and synthetic PII canaries. +- `benchmarks/truthbench/fixtures/healthcare.py` — healthcare frame and oracle labels for ICD/LOINC, temperature and dose units, FHIR/partial/impossible dates, Unicode, and PHI canaries. +- `benchmarks/truthbench/fixtures/retail.py` — retail frame and oracle labels for leading-zero identifiers, free/return quantities, locale currency formats, mojibake/HTML, multilingual values, and review PII. +- `benchmarks/truthbench/fixtures/crm.py` — CRM frame and oracle labels for Unicode/contact ambiguity, lifecycle contradiction, zero-width/hidden PII, Apple semantic traps, and protected IDs. +- `benchmarks/truthbench/fixtures/__init__.py` — registry entries and stable domain order. +- `tests/truthbench/test_fixtures.py` — domain completeness, disposition, deterministic-seed, and adversarial marker tests. + +## Self-review + +- Every builder emits exactly 16 rows with stable string indexes, fixed `2026-01-15` UTC reference metadata, explicit locale metadata, complete physical-cell labels, and at least 12 injected adversarial cells. +- All four dispositions are represented per domain. Row cases cover exact duplicates and removed rows; schema cases cover added, removed, renamed, reordered, and type-drifted columns without mislabeling absent cells. +- Sensitive values are whole-value synthetic `.invalid`, `TB-*`, or `555-01xx` forms, preserving the base builder's redaction and canary invariants. Zero-width examples are intentionally non-sensitive representation traps. +- Seed values are included only in a deterministic batch marker; same-seed builds produce byte-stable serialized oracle payloads and identical fixture hashes for seeds `1729` and `2718`. +- No FreshData runtime, LLM/provider, network, or external data dependency is used. + +## Concerns + +- The legacy `minimal` fixture remains registered for backwards compatibility; the four Task 4 domains are appended in stable order. A later registry task may choose to retire `minimal` once its callers migrate. +- Row/schema expectations are metadata cases (the physical frame remains rectangular), consistent with the oracle contract that removed rows/columns cannot have cell labels. + +## Review follow-up: healthcare reference codes and content assertions + +### RED + +After review, focused content tests were strengthened to inspect each actual `GoldCell.family`, disposition, and adversarial frame value. The first run exposed four failures: the healthcare rare-code test rejected the placeholder `G rare`; finance, retail, and CRM injection-count assertions correctly counted only non-preserve dispositions rather than all injected cells. + +```text +PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov +``` + +Result before fixes: `4 failed, 24 passed`. + +### GREEN + +Healthcare preserve cases now use values present in the bundled reference sets (`Z79.4`, `F17.210`, and `9843-4`), and tests load those references plus validate ICD/LOINC syntax. Content tests assert the finance, healthcare, retail, and CRM families directly against physical cells; injected-cell counts use non-background families, including preserve injections. + +```text +PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov +``` + +Result: `28 passed`. + +Complete adjacent verification (`tests/truthbench/test_fixtures.py`, `tests/truthbench/test_models_exact.py`, and `tests/truthbench/test_schema.py`) passed. Ruff check/format, mypy, and `git diff --check` also passed. + +## Review follow-up: contract-gap family coverage + +### RED + +Added direct assertions for the finance USD/EUR/INR conflict and zero-width memo, healthcare protected-DOB repair conflict and MRN tail canary, and exact row/schema family sets for finance, healthcare, and CRM. The initial run exposed the missing `zero-width-memo` family label (the value existed but was grouped under the broader invisible-PII family). + +```text +PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov +``` + +Result: `1 failed, 30 passed` (`StopIteration` while locating the required zero-width memo family). + +### GREEN + +The zero-width memo cell now has its own `zero-width-memo` family; all assertions inspect actual frame values, dispositions, sensitivity, and exact case-family sets. + +```text +PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov +``` + +Result: `31 passed`. + +Complete fixture/models/schema verification passed (`... passed`), as did Ruff check/format, mypy, and `git diff --check`. diff --git a/.superpowers/sdd/task-5-report.md b/.superpowers/sdd/task-5-report.md new file mode 100644 index 0000000..6ad9f93 --- /dev/null +++ b/.superpowers/sdd/task-5-report.md @@ -0,0 +1,85 @@ +# Task 5 Report: Complete eight-domain TruthBench corpus + +## Status + +DONE. Logistics, government, education, and insurance now have deterministic 16-row gold fixtures. The registry is the stable alphabetical eight-domain order (`crm`, `education`, `finance`, `government`, `healthcare`, `insurance`, `logistics`, `retail`) with the temporary minimal registry removed. + +## TDD evidence + +### RED + +After adding domain-content tests, the focused fixture suite failed with the expected missing-builder errors for all four new domains (`FixtureError: unknown fixture domain`). The failure run was: + +```text +PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov +``` + +### GREEN + +The focused suite now passes: + +```text +PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov +``` + +Result: `47 passed`. + +Adjacent fixture/model/schema verification: + +```text +PYTHONPATH=src python -m pytest \ + tests/truthbench/test_fixtures.py \ + tests/truthbench/test_models_exact.py \ + tests/truthbench/test_schema.py -q --no-cov +``` + +Result: `112 passed`. + +Static checks: + +```text +python -m ruff check benchmarks/truthbench/fixtures tests/truthbench/test_fixtures.py +python -m ruff format --check benchmarks/truthbench/fixtures tests/truthbench/test_fixtures.py +mypy --ignore-missing-imports --explicit-package-bases \ + benchmarks/truthbench/fixtures tests/truthbench/test_fixtures.py +git diff --check +``` + +All passed. + +## Coverage + +- `logistics.py` covers valid UN/LOCODE-like references, kg/lb and C/F units, cross-timezone windows, 24:00 transport values, address PII, late tracking, and protected shipment IDs. +- `government.py` covers leading-zero IDs, Indian/international grouping, fiscal/calendar ambiguity, multilingual labels, restricted national IDs, mixed legacy encoding, and retention/repair policy contradiction. +- `education.py` covers student IDs, letter/percentage/GPA scales, school-year ambiguity, zero scores, enrollment ordering, guardian contacts, FERPA notes, and protected grade-policy conflict. +- `insurance.py` covers policy/claim IDs, premium/reserve currency mismatch, negative reserve review, incident/report ordering, state contradiction, claimant/medical PII, and protected policy numbers. +- All four builders emit complete physical-cell labels, all four dispositions, row duplicate/removal cases, five schema drift cases, fixed UTC/reference metadata, deterministic seed batches, and privacy-safe synthetic canaries. +- Corpus-level tests assert all required trap categories occur across the eight domains. + +## Concerns + +None. No FreshData runtime, LLM/provider, network, or external data dependency is used. + +## Review follow-up: explicit contract coverage + +### RED + +The new required-family contract test intentionally used the repaired numeric output (`95`) as the adversarial frame value for education `edu-07`. The focused test failed because the actual frame value is the required raw value `"95%"` while the repair oracle separately stores `95.0` as its expected output. + +```text +PYTHONPATH=src python -m pytest \ + tests/truthbench/test_fixtures.py::test_required_domain_families_match_actual_values_and_dispositions \ + -q --no-cov +``` + +Observed: one failure at `edu-07` (`'95%' != 95`). + +### GREEN + +The contract now asserts the raw value, family, disposition, and typed repair output. It also uses an explicit per-domain family mapping covering every required category (including logistics lb/F units, government IDs/grouping/language/fiscal/protected case, education scales/contacts/protected grade, and insurance IDs/grouped premium/PII/protected policy). + +```text +PYTHONPATH=src python -m pytest tests/truthbench/test_fixtures.py -q --no-cov +``` + +Result: `48 passed`. diff --git a/.superpowers/sdd/task-6-report.md b/.superpowers/sdd/task-6-report.md new file mode 100644 index 0000000..ae4f228 --- /dev/null +++ b/.superpowers/sdd/task-6-report.md @@ -0,0 +1,202 @@ +# Task 6 Report: Privacy-safe values and exhaustive sink scanning + +## Status + +DONE. `SinkScanner` now scans normalized canary variants across nested TruthBench +sinks and emits only `Leak(canary_id, variant, path)` metadata. Redaction markers +contain run-scoped HMAC-SHA256 digests and never matched text. + +## TDD evidence + +### RED + +```text +PYTHONPATH=src python -m pytest tests/truthbench/test_privacy.py -q --no-cov +``` + +Observed collection failure before implementation: + +```text +ModuleNotFoundError: No module named 'benchmarks.truthbench.privacy' +``` + +### GREEN + +Focused privacy suite: + +```text +PYTHONPATH=src python -m pytest tests/truthbench/test_privacy.py -q --no-cov +``` + +Result: `18 passed`. + +Adjacent TruthBench suites: + +```text +PYTHONPATH=src python -m pytest \ + tests/truthbench/test_privacy.py \ + tests/truthbench/test_fixtures.py \ + tests/truthbench/test_models_exact.py \ + tests/truthbench/test_schema.py -q --no-cov +``` + +Result: `131 passed`. + +Static checks: + +```text +python -m ruff check benchmarks/truthbench/privacy.py \ + benchmarks/truthbench/__init__.py tests/truthbench/test_privacy.py +python -m ruff format --check benchmarks/truthbench/privacy.py \ + benchmarks/truthbench/__init__.py tests/truthbench/test_privacy.py +mypy --ignore-missing-imports --explicit-package-bases \ + benchmarks/truthbench/privacy.py tests/truthbench/test_privacy.py +git diff --check +``` + +## Review follow-up: MultiIndex and hostile-label privacy hardening + +### RED + +Added regression tests for MultiIndex columns/index level names, tuple-label +structure during redaction, and custom non-string labels whose stringification +contains a canary. Before the fix: + +```text +pytest -q tests/truthbench/test_privacy.py --no-cov +``` + +Result: `24 passed, 2 failed`. + +The failures were the expected defects: redacting tuple labels converted them +to lists and raised pandas `ValueError` (length mismatch), while hostile custom +labels were not scanned. + +### GREEN + +Structure-preserving label traversal/redaction now handles tuple/MultiIndex +levels and names, and sanitizes custom label paths before reporting or replacing +them with digest markers: + +```text +pytest -q tests/truthbench/test_privacy.py --no-cov +``` + +Result: `26 passed`. + +Adjacent TruthBench verification: + +```text +pytest -q tests/truthbench --no-cov +``` + +Result: `139 passed`. + +Static checks: + +```text +ruff check benchmarks/truthbench/privacy.py tests/truthbench/test_privacy.py +ruff format --check benchmarks/truthbench/privacy.py tests/truthbench/test_privacy.py +mypy benchmarks/truthbench/privacy.py +git diff --check +``` + +All passed. + +All passed. + +## Files + +- `benchmarks/truthbench/privacy.py` — `Leak`, `PrivacySafeValue`, named + normalizers, run-scoped HMAC scanner, recursive redaction, typed redaction + handling, pandas/dataclass/bytes support, self-test, and named sink entry points. +- `benchmarks/truthbench/__init__.py` — exports privacy scanner primitives. +- `tests/truthbench/test_privacy.py` — mutation matrix for every required + normalized form, nested sink coverage, redaction/self-test behavior, and typed + redaction digest safety. + +## Self-review + +- Leak objects contain only identifiers, transform labels, and JSONPath-like + locations; `repr(leaks)` cannot repeat canary text. +- Literal, case-folded, whitespace-stripped, punctuation-stripped, digit-only, + URL-decoded, HTML-unescaped, UTF-8/hex/escape bytes, NFKC/NFC/NFD, + zero-width-removed, and JSON-escaped forms are covered. +- Mapping, sequence, dataclass, pandas DataFrame/Series/Index, exception, + report, plan, generated-code, stream, markup, JSON, and failure-artifact sinks + are traversed. Exact redacted `TypedValue` payloads are treated as safe and + their HMAC digests are not re-scanned as plaintext. +- Redaction is recursive, emits `[REDACTED:]`, and `self_test` raises on + an unredacted result while accepting the scanner's own redacted output. + +## Concerns + +- The scanner intentionally treats digit-only normalized forms conservatively; + very short numeric canaries can match unrelated text. Fixtures use synthetic + identifiers and email/phone canaries, so this does not affect the bundled + corpus. +- Redacting a pandas object may change a sensitive column to object/string dtype, + which is preferable to retaining a raw value in an audit sink. + +## Follow-up RED/GREEN evidence + +A follow-up regression test used the exact sensitive `TypedValue` redaction shape +with a one-digit canary. Before the redacted-payload guard, the digest's hex text +was incorrectly reported as a digit-only leak. After adding the guard: + +```text +PYTHONPATH=src python -m pytest \ + tests/truthbench/test_privacy.py::test_scanner_accepts_exact_typed_redaction_without_scanning_digest \ + -q --no-cov +``` + +Result: `1 passed`. + +## Review follow-up: marker, key, and label hardening + +### RED + +The review regression matrix was run before the hardening changes: + +```text +PYTHONPATH=src python -m pytest tests/truthbench/test_privacy.py -q --no-cov +``` + +Result: `4 failed, 17 passed` for forged redaction markers, sensitive mapping +keys, pandas labels, and the digest compatibility alias. + +### GREEN + +After validating markers against the scanner's own 64-hex HMAC digest set, +redacting mapping keys and pandas column/index/name labels, scanning arbitrary +key stringifications, and adding `digest` as an alias: + +```text +PYTHONPATH=src python -m pytest tests/truthbench/test_privacy.py -q --no-cov +``` + +Result: `23 passed`. + +Adjacent TruthBench verification: + +```text +PYTHONPATH=src python -m pytest \ + tests/truthbench/test_privacy.py \ + tests/truthbench/test_fixtures.py \ + tests/truthbench/test_models_exact.py \ + tests/truthbench/test_schema.py -q --no-cov +``` + +Result: `136 passed`. + +Static checks were rerun after the follow-up and remained clean: + +```text +python -m ruff check benchmarks/truthbench/privacy.py \ + tests/truthbench/test_privacy.py +python -m ruff format --check benchmarks/truthbench/privacy.py \ + tests/truthbench/test_privacy.py +mypy --ignore-missing-imports --explicit-package-bases \ + benchmarks/truthbench/privacy.py tests/truthbench/test_privacy.py +git diff --check +``` diff --git a/benchmarks/truthbench/__init__.py b/benchmarks/truthbench/__init__.py new file mode 100644 index 0000000..d2e1690 --- /dev/null +++ b/benchmarks/truthbench/__init__.py @@ -0,0 +1,33 @@ +"""Deterministic oracle and result primitives for FreshData TruthBench.""" + +from .exact import TypedValue, canonical_json, encode_typed, exact_equal, stable_digest +from .models import ( + UNSET, + CaseExpectation, + DecisionRecord, + Disposition, + GateResult, + GoldCell, + RunResult, +) +from .privacy import Leak, PrivacySafeValue, SinkScanner, normalize_variants, redact_value + +__all__ = [ + "UNSET", + "CaseExpectation", + "DecisionRecord", + "Disposition", + "GateResult", + "GoldCell", + "RunResult", + "TypedValue", + "canonical_json", + "encode_typed", + "exact_equal", + "stable_digest", + "Leak", + "PrivacySafeValue", + "SinkScanner", + "normalize_variants", + "redact_value", +] diff --git a/benchmarks/truthbench/exact.py b/benchmarks/truthbench/exact.py new file mode 100644 index 0000000..98dacd3 --- /dev/null +++ b/benchmarks/truthbench/exact.py @@ -0,0 +1,254 @@ +from __future__ import annotations + +import hashlib +import hmac +import json +import math +from collections.abc import Mapping +from dataclasses import dataclass, field +from datetime import date, datetime, time, timedelta +from decimal import Decimal +from enum import Enum +from typing import Any + +import numpy as np +import pandas as pd + +JsonValue = Any + + +@dataclass(frozen=True) +class TypedValue: + """A scalar encoded without erasing its Python/pandas representation.""" + + kind: str + value: JsonValue + dtype: str | None + display: str + digest: str | None = None + redacted: bool = False + schema_version: int = field(default=1, init=False) + + @property + def type_label(self) -> str: + if self.dtype is None: + return self.kind + return f"{self.kind}[{self.dtype}]" + + def to_dict(self) -> dict[str, JsonValue]: + return { + "schema_version": self.schema_version, + "type": self.kind, + "dtype": self.dtype, + "value": self.value, + "display": self.display, + "digest": self.digest, + "redacted": self.redacted, + } + + @classmethod + def from_dict(cls, payload: Mapping[str, JsonValue]) -> TypedValue: + if payload.get("schema_version") != 1: + raise ValueError("unsupported typed value schema version") + return cls( + kind=str(payload["type"]), + dtype=None if payload["dtype"] is None else str(payload["dtype"]), + value=payload["value"], + display=str(payload["display"]), + digest=None if payload["digest"] is None else str(payload["digest"]), + redacted=bool(payload["redacted"]), + ) + + +def _dtype_name(dtype: Any, *, sensitive: bool) -> str | None: + if dtype is None: + return None + normalized = pd.api.types.pandas_dtype(dtype) + if isinstance(normalized, pd.CategoricalDtype): + categories = normalized.categories + if sensitive: + encoded_categories: JsonValue = {"count": len(categories)} + else: + encoded_categories = [encode_typed(value).to_dict() for value in categories] + category_json = canonical_json(encoded_categories) + ordered = "true" if normalized.ordered else "false" + return f"category[ordered={ordered};categories={category_json}]" + if isinstance(normalized, pd.StringDtype): + return f"string[{normalized.storage}]" + return str(normalized) + + +def _timezone_identity(value: Any) -> str | None: + if value is None: + return None + key = getattr(value, "key", None) + if isinstance(key, str): + return key + return str(value) + + +def _encode_scalar(value: Any) -> tuple[str, JsonValue, str]: + if value is pd.NA: + return "pandas.NA", None, "" + if value is pd.NaT: + return "pandas.NaT", None, "NaT" + if value is None: + return "python.none", None, "None" + + if isinstance(value, np.bool_): + scalar = bool(value) + return "numpy.bool_", scalar, str(scalar) + if isinstance(value, bool): + return "python.bool", value, str(value) + + if isinstance(value, np.integer): + return f"numpy.{value.dtype.name}", str(value), str(value) + if isinstance(value, int): + return "python.int", str(value), str(value) + + if isinstance(value, np.floating): + kind = f"numpy.{value.dtype.name}" + scalar = float(value) + if math.isnan(scalar): + return f"{kind}.nan", None, "NaN" + if math.isinf(scalar): + sign = "+" if scalar > 0 else "-" + return f"{kind}.infinity", sign, f"{sign}Infinity" + representation = repr(value.item()) + return kind, representation, representation + if isinstance(value, float): + if math.isnan(value): + return "python.float.nan", None, "NaN" + if math.isinf(value): + sign = "+" if value > 0 else "-" + return "python.float.infinity", sign, f"{sign}Infinity" + representation = repr(value) + return "python.float", representation, representation + + if isinstance(value, np.str_): + scalar = str(value) + return "numpy.str_", scalar, scalar + if isinstance(value, str): + return "python.str", value, value + + if isinstance(value, np.bytes_): + scalar = bytes(value).hex() + return "numpy.bytes_", scalar, repr(bytes(value)) + if isinstance(value, bytes): + return "python.bytes", value.hex(), repr(value) + + if isinstance(value, pd.Timestamp): + rendered = value.isoformat() + encoded = { + "iso": rendered, + "timezone": _timezone_identity(value.tz), + } + return "pandas.Timestamp", encoded, rendered + if isinstance(value, datetime): + rendered = value.isoformat() + encoded = {"iso": rendered, "timezone": _timezone_identity(value.tzinfo)} + return "python.datetime", encoded, rendered + if isinstance(value, date): + rendered = value.isoformat() + return "python.date", rendered, rendered + if isinstance(value, time): + rendered = value.isoformat() + encoded = {"iso": rendered, "timezone": _timezone_identity(value.tzinfo)} + return "python.time", encoded, rendered + + if isinstance(value, pd.Timedelta): + rendered = value.isoformat() + return "pandas.Timedelta", rendered, rendered + if isinstance(value, np.datetime64): + return f"numpy.{value.dtype}", str(value), str(value) + if isinstance(value, np.timedelta64): + return f"numpy.{value.dtype}", str(value), str(value) + if isinstance(value, timedelta): + rendered = repr(value) + return "python.timedelta", rendered, rendered + if isinstance(value, Decimal): + rendered = str(value) + return "python.Decimal", rendered, rendered + + raise TypeError(f"unsupported scalar type: {type(value).__name__}") + + +def encode_typed( + value: Any, + *, + dtype: Any = None, + sensitive: bool = False, + digest_key: bytes | None = None, +) -> TypedValue: + kind, encoded, display = _encode_scalar(value) + dtype_name = _dtype_name(dtype, sensitive=sensitive) + if not sensitive: + return TypedValue(kind=kind, value=encoded, dtype=dtype_name, display=display) + if digest_key is None: + raise ValueError("digest_key is required for sensitive values") + raw = TypedValue(kind=kind, value=encoded, dtype=dtype_name, display=display) + digest = hmac.new( + digest_key, + canonical_json(raw).encode("utf-8"), + hashlib.sha256, + ).hexdigest() + return TypedValue( + kind=kind, + value=None, + dtype=dtype_name, + display="[REDACTED]", + digest=digest, + redacted=True, + ) + + +def exact_equal( + left: Any, + right: Any, + *, + left_dtype: Any = None, + right_dtype: Any = None, +) -> bool: + return encode_typed(left, dtype=left_dtype) == encode_typed(right, dtype=right_dtype) + + +def _json_safe(value: Any, path: str = "$") -> JsonValue: + if isinstance(value, TypedValue): + return _json_safe(value.to_dict(), path) + if isinstance(value, Enum): + return _json_safe(value.value, path) + if value is None or isinstance(value, (bool, str)): + return value + if isinstance(value, int) and not isinstance(value, bool): + return value + if isinstance(value, float): + if not math.isfinite(value): + raise ValueError(f"{path} contains a non-finite number") + return value + if isinstance(value, Mapping): + result: dict[str, JsonValue] = {} + for key, item in value.items(): + if not isinstance(key, str): + raise TypeError(f"{path} contains an unsupported non-string key") + result[key] = _json_safe(item, f"{path}.{key}") + return result + if isinstance(value, (list, tuple)): + return [_json_safe(item, f"{path}[{index}]") for index, item in enumerate(value)] + raise TypeError(f"{path} contains unsupported type: {type(value).__name__}") + + +def canonical_json(value: Any) -> str: + return json.dumps( + _json_safe(value), + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + allow_nan=False, + ) + + +def stable_digest(value: Any, *, key: bytes) -> str: + if not isinstance(key, bytes): + raise TypeError("key must be bytes") + encoded = canonical_json(encode_typed(value)).encode("utf-8") + return hmac.new(key, encoded, hashlib.sha256).hexdigest() diff --git a/benchmarks/truthbench/fixtures/__init__.py b/benchmarks/truthbench/fixtures/__init__.py new file mode 100644 index 0000000..8ed5936 --- /dev/null +++ b/benchmarks/truthbench/fixtures/__init__.py @@ -0,0 +1,58 @@ +"""Fixture registry for the independent TruthBench oracle.""" + +from __future__ import annotations + +from typing import Callable + +from . import crm, education, finance, government, healthcare, insurance, logistics, retail +from .base import FixtureBuilder, FixtureError, TruthFixture + +_BUILDERS: dict[str, Callable[[int], TruthFixture]] = {} +DOMAINS: tuple[str, ...] = ( + "crm", + "education", + "finance", + "government", + "healthcare", + "insurance", + "logistics", + "retail", +) + + +def register_fixture(domain: str, builder: Callable[[int], TruthFixture]) -> None: + if not isinstance(domain, str) or not domain: + raise ValueError("fixture domain must be a non-empty string") + if domain in _BUILDERS: + raise ValueError(f"fixture domain already registered: {domain}") + _BUILDERS[domain] = builder + + +register_fixture("crm", crm.build) +register_fixture("education", education.build) +register_fixture("finance", finance.build) +register_fixture("government", government.build) +register_fixture("healthcare", healthcare.build) +register_fixture("insurance", insurance.build) +register_fixture("logistics", logistics.build) +register_fixture("retail", retail.build) + + +def build_fixture(domain: str, seed: int = 1729) -> TruthFixture: + try: + builder = _BUILDERS[domain] + except KeyError as exc: + raise FixtureError(f"unknown fixture domain: {domain!r}") from exc + fixture = builder(int(seed)) + fixture.validate() + return fixture + + +__all__ = [ + "DOMAINS", + "FixtureBuilder", + "FixtureError", + "TruthFixture", + "build_fixture", + "register_fixture", +] diff --git a/benchmarks/truthbench/fixtures/base.py b/benchmarks/truthbench/fixtures/base.py new file mode 100644 index 0000000..758a902 --- /dev/null +++ b/benchmarks/truthbench/fixtures/base.py @@ -0,0 +1,321 @@ +"""Independent, deterministic fixture construction primitives.""" + +from __future__ import annotations + +import hashlib +import re +from collections.abc import Callable, Mapping +from dataclasses import dataclass +from numbers import Integral +from typing import Any + +import pandas as pd + +from ..exact import canonical_json, encode_typed +from ..models import UNSET, CaseExpectation, Disposition, GoldCell + + +class FixtureError(ValueError): + """A fixture violates the TruthBench construction contract.""" + + +_HASH_KEY = b"truthbench-fixture-hash-v1" +_SYNTHETIC_EMAIL = re.compile(r"[A-Za-z0-9.!#$%&'*+/=?^_`{|}~-]+@[A-Za-z0-9.-]+\.invalid") +_SYNTHETIC_ID = re.compile(r"TB-[A-Za-z0-9-]+") +_SYNTHETIC_PHONE = re.compile(r"555[- .]?01[0-9]{2}") + + +def _synthetic_pii(value: Any) -> bool: + if isinstance(value, Integral) and not isinstance(value, bool): + value = str(value) + if not isinstance(value, str): + return False + return any( + pattern.fullmatch(value) for pattern in (_SYNTHETIC_EMAIL, _SYNTHETIC_ID, _SYNTHETIC_PHONE) + ) + + +def _typed(value: Any, dtype: Any = None, *, sensitive: bool = False) -> dict[str, Any]: + return encode_typed( + value, + dtype=dtype, + sensitive=sensitive, + digest_key=_HASH_KEY if sensitive else None, + ).to_dict() + + +@dataclass(frozen=True) +class TruthFixture: + """A pristine/adversarial pair and complete cell-level oracle.""" + + version: str + domain: str + seed: int + pristine: pd.DataFrame + frame: pd.DataFrame + cells: tuple[GoldCell, ...] + schema: Mapping[str, Any] + policy: Mapping[str, Any] + protected_columns: tuple[str, ...] + pii_canaries: Mapping[str, Any] + row_cases: tuple[CaseExpectation, ...] + schema_cases: tuple[CaseExpectation, ...] + fixture_hash: str + + @property + def adversarial(self) -> pd.DataFrame: + return self.frame + + @property + def pristine_frame(self) -> pd.DataFrame: + return self.pristine + + @property + def adversarial_frame(self) -> pd.DataFrame: + return self.frame + + def validate(self) -> None: + if not self.version or not self.domain: + raise FixtureError("fixture version and domain are required") + if not self.frame.index.is_unique or self.frame.index.hasnans: + raise FixtureError("frame row IDs must be unique and non-missing") + keys = [ + (str(row), str(column)) for row in self.frame.index for column in self.frame.columns + ] + labels = [(cell.row_id, cell.column) for cell in self.cells] + if len(labels) != len(set(labels)) or set(labels) != set(keys): + raise FixtureError("fixture must have exactly one label per physical cell") + cell_ids = [cell.cell_id for cell in self.cells] + if len(cell_ids) != len(set(cell_ids)): + raise FixtureError("fixture contains duplicate cell IDs") + for cell in self.cells: + if cell.fixture_version != self.version or cell.domain != self.domain: + raise FixtureError("cell identity does not match fixture") + expected_id = f"{self.version}:{self.domain}:{cell.row_id}:{cell.column}" + if cell.cell_id != expected_id: + raise FixtureError(f"cell {cell.cell_id} has an invalid stable ID") + if cell.disposition is Disposition.REPAIR and cell.expected_output is None: + raise FixtureError(f"repair cell {cell.cell_id} has no expected output") + if cell.sensitive and (not cell.canary_id or cell.canary_id not in self.pii_canaries): + raise FixtureError(f"sensitive cell {cell.cell_id} has no canary") + for case in (*self.row_cases, *self.schema_cases): + if case.domain != self.domain or case.fixture_version != self.version: + raise FixtureError("case identity does not match fixture") + if any(case.kind != "row" for case in self.row_cases): + raise FixtureError("row_cases must contain row expectations") + if any(case.kind != "schema" for case in self.schema_cases): + raise FixtureError("schema_cases must contain schema expectations") + if self.fixture_hash: + payload = self.to_dict() + payload.pop("fixture_hash", None) + expected_hash = hashlib.sha256(canonical_json(payload).encode("utf-8")).hexdigest() + if self.fixture_hash != expected_hash: + raise FixtureError("fixture hash does not match fixture contents") + + def to_dict(self) -> dict[str, Any]: + """Return a privacy-safe, JSON-ready representation.""" + + def frame_payload(frame: pd.DataFrame) -> list[dict[str, Any]]: + by_key = {(c.row_id, c.column): c for c in self.cells} + rows: list[dict[str, Any]] = [] + for row in frame.index: + row_id = str(row) + values = { + str(column): _typed( + frame.at[row, column], + frame[column].dtype, + sensitive=by_key[(row_id, str(column))].sensitive, + ) + for column in frame.columns + } + rows.append({"row_id": row_id, "values": values}) + return rows + + canaries = { + key: _typed(value, sensitive=True) for key, value in sorted(self.pii_canaries.items()) + } + return { + "version": self.version, + "domain": self.domain, + "seed": self.seed, + "pristine": frame_payload(self.pristine), + "frame": frame_payload(self.frame), + "cells": [cell.to_dict() for cell in self.cells], + "schema": dict(self.schema), + "policy": dict(self.policy), + "protected_columns": list(self.protected_columns), + "pii_canaries": canaries, + "row_cases": [case.to_dict() for case in self.row_cases], + "schema_cases": [case.to_dict() for case in self.schema_cases], + "fixture_hash": self.fixture_hash, + } + + +class FixtureBuilder: + """Build a fixture while maintaining a complete, atomic label matrix.""" + + def __init__( + self, + version: str, + domain: str, + frame: pd.DataFrame, + *, + seed: int = 1729, + schema: Mapping[str, Any] | None = None, + policy: Mapping[str, Any] | None = None, + protected_columns: tuple[str, ...] | list[str] = (), + pii_canaries: Mapping[str, Any] | None = None, + row_cases: tuple[CaseExpectation, ...] | list[CaseExpectation] = (), + schema_cases: tuple[CaseExpectation, ...] | list[CaseExpectation] = (), + ) -> None: + if not isinstance(frame, pd.DataFrame): + raise TypeError("frame must be a pandas DataFrame") + if frame.empty or frame.columns.empty: + raise FixtureError("frame must contain at least one row and column") + if not version or not domain: + raise FixtureError("version and domain are required") + if frame.index.hasnans or not frame.index.is_unique: + raise FixtureError("row IDs must be unique and non-missing") + row_ids = [str(row) for row in frame.index] + if any(not row_id for row_id in row_ids) or len(row_ids) != len(set(row_ids)): + raise FixtureError("row IDs must be unique and non-missing") + if any(not isinstance(column, str) or not column for column in frame.columns): + raise FixtureError("column names must be non-empty strings") + if len(set(frame.columns)) != len(frame.columns): + raise FixtureError("column names must be unique") + + self.version = version + self.domain = domain + self.seed = seed + self.pristine = frame.copy(deep=True) + self.pristine.index = row_ids + self.frame = frame.copy(deep=True) + self.frame.index = row_ids + self.schema = dict( + schema + or { + "columns": list(frame.columns), + "dtypes": {str(c): str(frame[c].dtype) for c in frame.columns}, + } + ) + self.policy = dict(policy or {}) + self.protected_columns = tuple(protected_columns) + unknown_protected = set(self.protected_columns) - set(frame.columns) + if unknown_protected: + raise FixtureError(f"unknown protected columns: {sorted(unknown_protected)}") + self._labels = { + (row_id, str(column)): GoldCell.create( + version, domain, row_id, str(column), Disposition.PRESERVE, family="background" + ) + for row_id in row_ids + for column in frame.columns + } + self._canaries: dict[str, Any] = dict(pii_canaries or {}) + if any(not _synthetic_pii(value) for value in self._canaries.values()): + raise FixtureError("PII canaries must use synthetic domains") + self._row_cases = list(row_cases) + self._schema_cases = list(schema_cases) + + def inject( + self, + row_id: str, + column: str, + value: Any, + disposition: Disposition | str, + *, + expected: Any = UNSET, + expected_dtype: Any = None, + family: str, + sensitive: bool = False, + ) -> None: + key = (str(row_id), str(column)) + if key not in self._labels: + raise FixtureError(f"unknown cell {key}") + if not isinstance(family, str) or not family: + raise FixtureError("family is required") + try: + parsed = Disposition(disposition) + except ValueError as exc: + raise FixtureError(f"unknown disposition: {disposition!r}") from exc + if parsed is Disposition.REPAIR and expected is UNSET: + raise FixtureError("repair cells require an expected output") + if sensitive and not _synthetic_pii(value): + raise FixtureError("sensitive values must use synthetic PII domains") + + canary_id = None + if sensitive: + canary_id = f"{self.domain}-{key[0]}-{key[1]}" + candidate = GoldCell.create( + self.version, + self.domain, + key[0], + key[1], + parsed, + expected_output=expected, + expected_dtype=expected_dtype, + family=family, + sensitive=sensitive, + canary_id=canary_id, + ) + previous = self._labels[key] + if ( + previous.disposition is Disposition.REPAIR + and candidate.disposition is Disposition.REPAIR + and previous.expected_output != candidate.expected_output + ): + raise FixtureError(f"contradictory repair output for cell {key}") + + # All validation above happens before either mutable structure changes. + if previous.canary_id and previous.canary_id != canary_id: + self._canaries.pop(previous.canary_id, None) + if canary_id: + self._canaries[canary_id] = value + # A corruption may intentionally change a column's scalar type. Cast + # to object first so pandas does not coerce or emit a FutureWarning. + self.frame[key[1]] = self.frame[key[1]].astype(object) + self.frame.at[key[0], key[1]] = value + self._labels[key] = candidate + + def add_row_case(self, name: str, disposition: Disposition | str, **kwargs: Any) -> None: + self._row_cases.append( + CaseExpectation.create(self.version, self.domain, "row", name, disposition, **kwargs) + ) + + def add_schema_case(self, name: str, disposition: Disposition | str, **kwargs: Any) -> None: + self._schema_cases.append( + CaseExpectation.create( + self.version, self.domain, "schema", name, disposition, **kwargs + ) + ) + + def build(self) -> TruthFixture: + cells = tuple( + self._labels[(str(row), str(column))] + for row in self.frame.index + for column in self.frame.columns + ) + fixture = TruthFixture( + version=self.version, + domain=self.domain, + seed=self.seed, + pristine=self.pristine.copy(deep=True), + frame=self.frame.copy(deep=True), + cells=cells, + schema=dict(self.schema), + policy=dict(self.policy), + protected_columns=self.protected_columns, + pii_canaries=dict(self._canaries), + row_cases=tuple(self._row_cases), + schema_cases=tuple(self._schema_cases), + fixture_hash="", + ) + fixture.validate() + payload = fixture.to_dict() + payload.pop("fixture_hash", None) + digest = hashlib.sha256(canonical_json(payload).encode("utf-8")).hexdigest() + result = TruthFixture(**{**fixture.__dict__, "fixture_hash": digest}) + result.validate() + return result + + +BuilderFactory = Callable[[int], TruthFixture] diff --git a/benchmarks/truthbench/fixtures/crm.py b/benchmarks/truthbench/fixtures/crm.py new file mode 100644 index 0000000..6878d17 --- /dev/null +++ b/benchmarks/truthbench/fixtures/crm.py @@ -0,0 +1,257 @@ +"""Deterministic CRM gold fixture for TruthBench.""" + +from __future__ import annotations + +import pandas as pd + +from ..models import Disposition +from .base import FixtureBuilder, TruthFixture + + +def build(seed: int = 1729) -> TruthFixture: + rows = [f"crm-{i:02d}" for i in range(1, 17)] + frame = pd.DataFrame( + { + "customer_id": [f"TB-CRM-{i:04d}" for i in range(1, 17)], + "first_name": [ + "Ana", + "Jose\u0301", + "Miyuki", + "Zoë", + "Liam", + "Nia", + "Omar", + "Riya", + "Noel", + "Saanvi", + "Evan", + "Ivy", + "Kai", + "Mara", + "Theo", + "Uma", + ], + "last_name": [ + "Stone", + "García", + "Chen", + "Müller", + "Jones", + "Patel", + "Khan", + "Roy", + "Smith", + "Das", + "Lee", + "Brown", + "Cruz", + "Mori", + "Cole", + "Singh", + ], + "email": [ + "ana@example.invalid", + "jose@example.invalid", + "miyuki@example.invalid", + "zoe@example.invalid", + "liam@example.invalid", + "nia@example.invalid", + "omar@example.invalid", + "riya@example.invalid", + "noel@example.invalid", + "saanvi@example.invalid", + "evan@example.invalid", + "ivy@example.invalid", + "kai@example.invalid", + "mara@example.invalid", + "theo@example.invalid", + "uma@example.invalid", + ], + "phone": ["555-0101"] * 16, + "country": [ + "US", + "DE", + "IN", + "GB", + "US", + "CA", + "AU", + "JP", + "BR", + "FR", + "US", + "IN", + "MX", + "ES", + "ZA", + "SG", + ], + "language": [ + "en", + "de", + "hi", + "en", + "en", + "fr", + "en", + "ja", + "pt", + "fr", + "en", + "hi", + "es", + "es", + "en", + "en", + ], + "signup_date": ["2026-01-15"] * 16, + "lifecycle": [ + "lead", + "prospect", + "customer", + "churned", + "lead", + "prospect", + "customer", + "lead", + "prospect", + "customer", + "lead", + "prospect", + "customer", + "lead", + "prospect", + "customer", + ], + "lead_source": [ + "web", + "referral", + "apple", + "partner", + "web", + "event", + "apple", + "search", + "web", + "partner", + "event", + "web", + "referral", + "search", + "web", + "event", + ], + "employer": [ + "Acme", + "Globex", + "Apple", + "Northstar", + "Acme", + "Cedar", + "Apple", + "Delta", + "Evergreen", + "Futura", + "Globex", + "Helios", + "Apple", + "Ion", + "Juniper", + "Kappa", + ], + "notes": ["ordinary"] * 16, + "batch": [f"seed-{int(seed)}"] * 16, + }, + index=rows, + ) + builder = FixtureBuilder( + "v1", + "crm", + frame, + seed=int(seed), + schema={ + "columns": list(frame.columns), + "dtypes": {c: str(frame[c].dtype) for c in frame.columns}, + }, + policy={ + "reference_date": "2026-01-15", + "timezone": "UTC", + "locale": "en_US", + "supported_countries": ["US", "DE", "IN", "GB"], + "supported_languages": ["en", "de", "hi", "ja"], + "protected_customer_id_policy": "preserve", + }, + protected_columns=("customer_id",), + ) + builder.inject( + "crm-02", + "first_name", + "Jose\u0301", + Disposition.REPAIR, + expected="José", + family="combining-unicode-name", + ) + builder.inject( + "crm-03", + "email", + "miyuki@example.invalid", + Disposition.PRESERVE, + family="reserved-invalid-email", + sensitive=True, + ) + builder.inject( + "crm-04", + "phone", + "555 0101", + Disposition.REPAIR, + expected="555-0101", + family="spaced-phone", + ) + builder.inject("crm-05", "country", "US/CA", Disposition.REVIEW, family="ambiguous-country") + builder.inject("crm-06", "language", "en/fr", Disposition.REVIEW, family="ambiguous-language") + builder.inject( + "crm-07", "signup_date", "01/02/2025", Disposition.REVIEW, family="ambiguous-date" + ) + builder.inject( + "crm-08", "lifecycle", "lead|churned", Disposition.REVIEW, family="lifecycle-contradiction" + ) + builder.inject( + "crm-09", + "email", + "TB-CRM-ZW-EMAIL", + Disposition.FLAG, + family="zero-width-email", + sensitive=True, + ) + builder.inject( + "crm-11", + "email", + "lead\u200b@example.invalid", + Disposition.FLAG, + family="zero-width-email", + ) + builder.inject( + "crm-10", "notes", "TB-CRM-SSN-0001", Disposition.FLAG, family="hidden-ssn", sensitive=True + ) + builder.inject( + "crm-03", "lead_source", "apple", Disposition.PRESERVE, family="apple-lead-source" + ) + builder.inject("crm-03", "employer", "Apple", Disposition.PRESERVE, family="apple-employer") + builder.inject( + "crm-16", + "customer_id", + "TB-CRM-CUSTOMER-TAIL", + Disposition.REVIEW, + family="protected-customer-id", + sensitive=True, + ) + builder.add_row_case( + "exact-duplicate-crm-02-crm-03", Disposition.FLAG, family="exact-duplicate" + ) + builder.add_row_case("removed-crm-15", Disposition.REVIEW, family="removed-row") + builder.add_schema_case("added-column", Disposition.REVIEW, family="added-column") + builder.add_schema_case("removed-column", Disposition.REVIEW, family="removed-column") + builder.add_schema_case("renamed-column", Disposition.REVIEW, family="renamed-column") + builder.add_schema_case("reordered-columns", Disposition.REVIEW, family="reordered-columns") + builder.add_schema_case("type-drifted-phone", Disposition.REVIEW, family="type-drifted-column") + return builder.build() diff --git a/benchmarks/truthbench/fixtures/education.py b/benchmarks/truthbench/fixtures/education.py new file mode 100644 index 0000000..039a055 --- /dev/null +++ b/benchmarks/truthbench/fixtures/education.py @@ -0,0 +1,187 @@ +"""Deterministic education gold fixture for TruthBench.""" + +from __future__ import annotations + +import pandas as pd + +from ..models import Disposition +from .base import FixtureBuilder, TruthFixture + + +def build(seed: int = 1729) -> TruthFixture: + rows = [f"edu-{i:02d}" for i in range(1, 17)] + frame = pd.DataFrame( + { + "student_id": [ + "000123", + "000124", + "000125", + "000126", + "000127", + "000128", + "000129", + "000130", + "000131", + "000132", + "000133", + "000134", + "000135", + "000136", + "000137", + "000138", + ], + "grade_letter": [ + "A", + "A-", + "B+", + "B", + "C+", + "C", + "B-", + "A", + "A", + "B", + "C", + "D", + "A", + "B+", + "C", + "A", + ], + "score_percent": [ + "95", + "90", + "87.5", + "82", + "78", + "72", + "68", + "100", + "0", + "88", + "76", + "61", + "93", + "86", + "74", + "96", + ], + "gpa": [ + 4.0, + 3.7, + 3.3, + 3.0, + 2.7, + 2.3, + 2.0, + 4.0, + 0.0, + 3.3, + 2.7, + 1.7, + 3.9, + 3.3, + 2.3, + 4.0, + ], + "school_year": ["2025-2026"] * 16, + "enrollment_date": ["2025-08-15"] * 16, + "completion_date": ["2026-05-30"] * 16, + "guardian_email": ["guardian1@example.invalid"] * 16, + "guardian_phone": ["555-0101"] * 16, + "ferpa_notes": ["routine"] * 16, + "grade_policy": ["A>=90;B>=80;C>=70"] * 16, + "language": ["en"] * 16, + "batch": [f"seed-{int(seed)}"] * 16, + }, + index=rows, + ) + builder = FixtureBuilder( + "v1", + "education", + frame, + seed=int(seed), + schema={ + "columns": list(frame.columns), + "dtypes": {c: str(frame[c].dtype) for c in frame.columns}, + }, + policy={ + "reference_date": "2026-01-15", + "timezone": "UTC", + "locale": "en_US", + "grade_scales": ["letter", "percentage", "GPA"], + "protected_grade_policy": "preserve", + }, + protected_columns=("student_id", "grade_letter"), + ) + builder.inject( + "edu-01", "student_id", "000123", Disposition.PRESERVE, family="leading-zero-student-id" + ) + builder.inject( + "edu-02", "grade_letter", "A-", Disposition.PRESERVE, family="letter-grade-scale" + ) + builder.inject("edu-03", "score_percent", 0, Disposition.PRESERVE, family="zero-score") + builder.inject( + "edu-04", "school_year", "2025/26", Disposition.REVIEW, family="school-year-ambiguity" + ) + builder.inject( + "edu-05", + "enrollment_date", + "2026-02-01", + Disposition.REVIEW, + family="enrollment-date-ordering", + ) + builder.inject( + "edu-06", + "completion_date", + "2025-01-01", + Disposition.REVIEW, + family="enrollment-date-ordering", + ) + builder.inject( + "edu-07", + "score_percent", + "95%", + Disposition.REPAIR, + expected=95.0, + expected_dtype="float64", + family="percentage-scale", + ) + builder.inject("edu-08", "gpa", 4.0, Disposition.PRESERVE, family="gpa-scale") + builder.inject( + "edu-09", + "guardian_email", + "guardian@example.invalid", + Disposition.FLAG, + family="guardian-contact-pii", + sensitive=True, + ) + builder.inject( + "edu-10", + "guardian_phone", + "555-0110", + Disposition.FLAG, + family="guardian-contact-pii", + sensitive=True, + ) + builder.inject( + "edu-10", + "ferpa_notes", + "TB-EDU-FERPA-0001", + Disposition.FLAG, + family="ferpa-sensitive-notes", + sensitive=True, + ) + builder.inject( + "edu-16", "grade_letter", "A", Disposition.REVIEW, family="protected-grade-policy-conflict" + ) + builder.add_row_case( + "exact-duplicate-edu-02-edu-03", Disposition.FLAG, family="exact-duplicate" + ) + builder.add_row_case("removed-edu-15", Disposition.REVIEW, family="removed-row") + builder.add_schema_case("added-column", Disposition.REVIEW, family="added-column") + builder.add_schema_case("removed-column", Disposition.REVIEW, family="removed-column") + builder.add_schema_case("renamed-column", Disposition.REVIEW, family="renamed-column") + builder.add_schema_case("reordered-columns", Disposition.REVIEW, family="reordered-columns") + builder.add_schema_case("type-drifted-score", Disposition.REVIEW, family="type-drifted-column") + return builder.build() diff --git a/benchmarks/truthbench/fixtures/finance.py b/benchmarks/truthbench/fixtures/finance.py new file mode 100644 index 0000000..511b41c --- /dev/null +++ b/benchmarks/truthbench/fixtures/finance.py @@ -0,0 +1,204 @@ +"""Deterministic finance gold fixture for TruthBench.""" + +from __future__ import annotations + +import pandas as pd + +from ..models import Disposition +from .base import FixtureBuilder, TruthFixture + + +def build(seed: int = 1729) -> TruthFixture: + rows = [f"fin-{i:02d}" for i in range(1, 17)] + frame = pd.DataFrame( + { + "account_id": [f"TB-FIN-{i:04d}" for i in range(1, 17)], + "asset": [ + "bond", + "fund", + "apple", + "stock", + "cash", + "fx", + "etf", + "bond", + "fund", + "stock", + "cash", + "fx", + "etf", + "bond", + "fund", + "stock", + ], + "company": [ + "Acme Bank", + "Northstar", + "Acme", + "Apple", + "Cedar", + "Delta", + "Evergreen", + "Futura", + "Globex", + "Helios", + "Ion", + "Juniper", + "Kappa", + "Lumen", + "Mosaic", + "Nadir", + ], + "ticker": [ + "ACME", + "NSTAR", + "APPL", + "AAPL", + "CEDR", + "DLTA", + "EVER", + "FUTR", + "GLOB", + "HELI", + "ION", + "JUN", + "KAPP", + "LUME", + "MOSC", + "NADI", + ], + "price": [ + 10.25, + 0.00, + "apple", + 155.20, + -4.5, + 9999999999.99, + "₹1,23,456.70", + 123.45, + 9.99, + 42.0, + 18.75, + 21.0, + 33.0, + 7.5, + 8.25, + 12.0, + ], + "currency": [ + "USD", + "USD", + "USD", + "USD", + "EUR", + "USD", + "INR", + "EUR", + "USD", + "USD", + "USD", + "EUR", + "INR", + "USD", + "USD", + "USD", + ], + "trade_date": ["2026-01-15"] * 16, + "memo": ["ordinary"] * 16, + "balance": [ + 100.0, + 200.0, + 300.0, + 400.0, + 500.0, + 600.0, + 700.0, + 800.0, + 900.0, + 1000.0, + 1100.0, + 1200.0, + 1300.0, + 1400.0, + 1500.0, + 1600.0, + ], + "locale": ["en_US"] * 16, + "batch": [f"seed-{int(seed)}"] * 16, + }, + index=rows, + ) + builder = FixtureBuilder( + "v1", + "finance", + frame, + seed=int(seed), + schema={ + "columns": list(frame.columns), + "dtypes": {c: str(frame[c].dtype) for c in frame.columns}, + }, + policy={ + "reference_date": "2026-01-15", + "timezone": "UTC", + "locale": "en_US", + "currency": "USD", + "supported_currencies": ["USD", "EUR", "INR"], + "protected_ticker_policy": "preserve", + }, + protected_columns=("account_id", "ticker"), + ) + builder.inject("fin-03", "price", "apple", Disposition.REVIEW, family="semantic-apple-price") + builder.inject("fin-04", "company", "Apple", Disposition.PRESERVE, family="apple-company") + builder.inject( + "fin-04", "ticker", "AAPL", Disposition.REVIEW, family="protected-ticker-policy-conflict" + ) + builder.inject("fin-02", "price", "0.00", Disposition.PRESERVE, family="zero-price") + builder.inject("fin-05", "price", -4.5, Disposition.FLAG, family="negative-value") + builder.inject("fin-06", "price", 9999999999.99, Disposition.FLAG, family="extreme-value") + builder.inject( + "fin-07", + "price", + "₹1,23,456.70", + Disposition.REPAIR, + expected=123456.70, + expected_dtype="float64", + family="indian-grouped-currency", + ) + builder.inject("fin-08", "currency", "EUR", Disposition.REVIEW, family="usd-eur-inr-conflict") + builder.inject("fin-09", "currency", "INR", Disposition.REVIEW, family="usd-eur-inr-conflict") + builder.inject( + "fin-10", "trade_date", "01/02/2025", Disposition.REVIEW, family="ambiguous-date-format" + ) + builder.inject( + "fin-11", + "memo", + "tb.finance+memo@example.invalid", + Disposition.FLAG, + family="invisible-pii-memo", + sensitive=True, + ) + builder.inject( + "fin-12", + "memo", + "tb.finance+hidden@example.invalid\u200b", + Disposition.FLAG, + family="zero-width-memo", + ) + builder.inject( + "fin-16", + "account_id", + "TB-FIN-ACCOUNT-TAIL", + Disposition.FLAG, + family="tail-row-account-canary", + sensitive=True, + ) + builder.add_row_case( + "exact-duplicate-fin-02-fin-03", Disposition.FLAG, family="exact-duplicate" + ) + builder.add_row_case("removed-fin-15", Disposition.REVIEW, family="removed-row") + builder.add_schema_case("added-column", Disposition.REVIEW, family="added-column") + builder.add_schema_case("removed-column", Disposition.REVIEW, family="removed-column") + builder.add_schema_case("renamed-column", Disposition.REVIEW, family="renamed-column") + builder.add_schema_case("reordered-columns", Disposition.REVIEW, family="reordered-columns") + builder.add_schema_case("type-drifted-price", Disposition.REVIEW, family="type-drifted-column") + return builder.build() diff --git a/benchmarks/truthbench/fixtures/government.py b/benchmarks/truthbench/fixtures/government.py new file mode 100644 index 0000000..14dc0e2 --- /dev/null +++ b/benchmarks/truthbench/fixtures/government.py @@ -0,0 +1,170 @@ +"""Deterministic government gold fixture for TruthBench.""" + +from __future__ import annotations + +import pandas as pd + +from ..models import Disposition +from .base import FixtureBuilder, TruthFixture + + +def build(seed: int = 1729) -> TruthFixture: + rows = [f"gov-{i:02d}" for i in range(1, 17)] + frame = pd.DataFrame( + { + "district_id": [ + "001", + "002", + "003", + "004", + "005", + "006", + "007", + "008", + "009", + "010", + "011", + "012", + "013", + "014", + "015", + "016", + ], + "case_id": [ + "000001", + "000123", + "000003", + "000004", + "000005", + "000006", + "000007", + "000008", + "000009", + "000010", + "000011", + "000012", + "000013", + "000014", + "000015", + "000016", + ], + "agency": [ + "Revenue Department", + "भारत सरकार / Government of India", + "Ministry of Health", + "City Council", + "税務局", + "Prefecture Office", + "State Secretariat", + "Public Works", + "Education Board", + "Transport Authority", + "Treasury", + "Registry", + "Civil Court", + "Labor Office", + "Statistics Bureau", + "Revenue Department", + ], + "fiscal_year": ["2026"] * 16, + "calendar_year": ["2026"] * 16, + "language": ["en"] * 16, + "notes": ["routine"] * 16, + "encoding": ["UTF-8"] * 16, + "retention_policy": ["retain 7 years"] * 16, + "repair_policy": ["repair after 30 days"] * 16, + "batch": [f"seed-{int(seed)}"] * 16, + }, + index=rows, + ) + builder = FixtureBuilder( + "v1", + "government", + frame, + seed=int(seed), + schema={ + "columns": list(frame.columns), + "dtypes": {c: str(frame[c].dtype) for c in frame.columns}, + }, + policy={ + "reference_date": "2026-01-15", + "timezone": "UTC", + "locale": "en_US", + "supported_languages": ["en", "hi", "de", "zh"], + "protected_case_policy": "preserve", + }, + protected_columns=("district_id", "case_id"), + ) + builder.inject( + "gov-01", "district_id", "007", Disposition.PRESERVE, family="leading-zero-district-id" + ) + builder.inject( + "gov-02", "case_id", "000123", Disposition.PRESERVE, family="leading-zero-case-id" + ) + builder.inject( + "gov-03", + "agency", + "भारत सरकार / Government of India", + Disposition.PRESERVE, + family="indian-international-grouping", + ) + builder.inject( + "gov-05", + "fiscal_year", + "2025-26", + Disposition.REVIEW, + family="fiscal-calendar-year-conflict", + ) + builder.inject( + "gov-06", + "encoding", + "Café", + Disposition.REPAIR, + expected="Café", + family="mixed-legacy-encoding", + ) + builder.inject( + "gov-07", "language", "हिन्दी / English", Disposition.PRESERVE, family="multilingual-agency" + ) + builder.inject( + "gov-09", + "notes", + "TB-GOV-NATIONAL-ID-0001", + Disposition.FLAG, + family="restricted-national-id", + sensitive=True, + ) + builder.inject( + "gov-11", + "retention_policy", + "retain 7 years", + Disposition.REVIEW, + family="contradictory-retention-repair-policy", + ) + builder.inject( + "gov-11", + "repair_policy", + "repair after 30 days", + Disposition.REVIEW, + family="contradictory-retention-repair-policy", + ) + builder.inject( + "gov-16", + "case_id", + "TB-GOV-CASE-TAIL", + Disposition.REVIEW, + family="protected-case-id-conflict", + sensitive=True, + ) + builder.add_row_case( + "exact-duplicate-gov-02-gov-03", Disposition.FLAG, family="exact-duplicate" + ) + builder.add_row_case("removed-gov-15", Disposition.REVIEW, family="removed-row") + builder.add_schema_case("added-column", Disposition.REVIEW, family="added-column") + builder.add_schema_case("removed-column", Disposition.REVIEW, family="removed-column") + builder.add_schema_case("renamed-column", Disposition.REVIEW, family="renamed-column") + builder.add_schema_case("reordered-columns", Disposition.REVIEW, family="reordered-columns") + builder.add_schema_case( + "type-drifted-fiscal-year", Disposition.REVIEW, family="type-drifted-column" + ) + return builder.build() diff --git a/benchmarks/truthbench/fixtures/healthcare.py b/benchmarks/truthbench/fixtures/healthcare.py new file mode 100644 index 0000000..7ddeed2 --- /dev/null +++ b/benchmarks/truthbench/fixtures/healthcare.py @@ -0,0 +1,206 @@ +"""Deterministic healthcare gold fixture for TruthBench.""" + +from __future__ import annotations + +import pandas as pd + +from ..models import Disposition +from .base import FixtureBuilder, TruthFixture + + +def build(seed: int = 1729) -> TruthFixture: + rows = [f"hc-{i:02d}" for i in range(1, 17)] + frame = pd.DataFrame( + { + "mrn": [f"TB-HC-{i:04d}" for i in range(1, 17)], + "patient_name": [ + "Amina Khan", + "Luca Rossi", + "Marta Silva", + "Noah Reed", + "Iris Chen", + "Sofia Patel", + "Eli Jones", + "Ravi Singh", + "Nora Stone", + "Mina Park", + "Owen Hall", + "Zoe King", + "Leo Cruz", + "Aya Mori", + "Finn Cole", + "Uma Roy", + ], + "diagnosis_code": [ + "R69", + "E11.9", + "I10", + "Z79.4", + "J45.9", + "M54.5", + "N39.0", + "K21.9", + "L20.9", + "D50.9", + "G43.9", + "H25.1", + "F17.210", + "B20", + "A09", + "Z00.0", + ], + "loinc": [ + "LP21258-6", + "2345-7", + "718-7", + "9843-4", + "4548-4", + "8310-5", + "2951-2", + "2160-0", + "6690-2", + "26499-4", + "3094-0", + "33747-0", + "9843-4", + "1751-7", + "2823-3", + "1975-2", + ], + "temperature": [ + 36.8, + 37.0, + 36.5, + 98.6, + 37.1, + 36.9, + 37.2, + 36.7, + 36.6, + 37.0, + 36.8, + 36.9, + 37.1, + 36.8, + 36.7, + 36.9, + ], + "dose": [ + "5 mg", + "10 mg", + "2.5 mg", + "5000 mcg", + "1 mg", + "20 mg", + "5 mg", + "2 mg", + "8 mg", + "4 mg", + "3 mg", + "6 mg", + "7 mg", + "9 mg", + "1 mg", + "2 mg", + ], + "event_date": ["2026-01-15"] * 16, + "dob": ["1980-01-01"] * 16, + "notes": ["routine"] * 16, + "phone": ["555-0101"] * 16, + "language": ["en"] * 16, + "batch": [f"seed-{int(seed)}"] * 16, + }, + index=rows, + ) + builder = FixtureBuilder( + "v1", + "healthcare", + frame, + seed=int(seed), + schema={ + "columns": list(frame.columns), + "dtypes": {c: str(frame[c].dtype) for c in frame.columns}, + }, + policy={ + "reference_date": "2026-01-15", + "timezone": "UTC", + "locale": "en_US", + "standards": ["FHIR", "ICD-10", "LOINC"], + "temperature_unit": "C", + "protected_dob_policy": "preserve", + }, + protected_columns=("mrn", "dob"), + ) + builder.inject( + "hc-04", "diagnosis_code", "Z79.4", Disposition.PRESERVE, family="rare-icd-valid" + ) + builder.inject("hc-04", "loinc", "9843-4", Disposition.PRESERVE, family="rare-loinc-valid") + builder.inject( + "hc-04", "temperature", 98.6, Disposition.REVIEW, family="celsius-fahrenheit-conflict" + ) + builder.inject("hc-01", "dose", "5 mg", Disposition.PRESERVE, family="dose-unit-valid") + builder.inject( + "hc-04", + "dose", + "5000 mcg", + Disposition.REPAIR, + expected=5.0, + expected_dtype="float64", + family="mg-mcg-unit-conversion", + ) + builder.inject("hc-05", "event_date", "2025-01", Disposition.REVIEW, family="partial-date") + builder.inject( + "hc-06", + "event_date", + "2025-01-15T12:00:00Z", + Disposition.REPAIR, + expected="2025-01-15", + family="fhir-date", + ) + builder.inject("hc-07", "event_date", "2025-02-30", Disposition.FLAG, family="impossible-date") + builder.inject( + "hc-08", + "patient_name", + "Jose\u0301", + Disposition.REPAIR, + expected="José", + family="decomposed-unicode", + ) + builder.inject( + "hc-09", "notes", "TB-HC-PHI-0001", Disposition.FLAG, family="phi-in-notes", sensitive=True + ) + builder.inject( + "hc-10", + "phone", + "555-0107", + Disposition.PRESERVE, + family="synthetic-phone", + sensitive=True, + ) + builder.inject( + "hc-11", "notes", "TB-HC-MRN-NOTE", Disposition.FLAG, family="mrn-in-notes", sensitive=True + ) + builder.inject( + "hc-12", "notes", "555-0112", Disposition.FLAG, family="phone-in-notes", sensitive=True + ) + builder.inject( + "hc-16", + "mrn", + "TB-HC-MRN-TAIL", + Disposition.REVIEW, + family="mrn-tail-canary", + sensitive=True, + ) + builder.inject( + "hc-15", "dob", "01/01/1980", Disposition.REVIEW, family="protected-dob-repair-conflict" + ) + builder.add_row_case("exact-duplicate-hc-02-hc-03", Disposition.FLAG, family="exact-duplicate") + builder.add_row_case("removed-hc-15", Disposition.REVIEW, family="removed-row") + builder.add_schema_case("added-column", Disposition.REVIEW, family="added-column") + builder.add_schema_case("removed-column", Disposition.REVIEW, family="removed-column") + builder.add_schema_case("renamed-column", Disposition.REVIEW, family="renamed-column") + builder.add_schema_case("reordered-columns", Disposition.REVIEW, family="reordered-columns") + builder.add_schema_case( + "type-drifted-temperature", Disposition.REVIEW, family="type-drifted-column" + ) + return builder.build() diff --git a/benchmarks/truthbench/fixtures/insurance.py b/benchmarks/truthbench/fixtures/insurance.py new file mode 100644 index 0000000..25ce595 --- /dev/null +++ b/benchmarks/truthbench/fixtures/insurance.py @@ -0,0 +1,208 @@ +"""Deterministic insurance gold fixture for TruthBench.""" + +from __future__ import annotations + +import pandas as pd + +from ..models import Disposition +from .base import FixtureBuilder, TruthFixture + + +def build(seed: int = 1729) -> TruthFixture: + rows = [f"ins-{i:02d}" for i in range(1, 17)] + frame = pd.DataFrame( + { + "policy_number": [ + "00012345", + "00012346", + "00012347", + "00012348", + "00012349", + "00012350", + "00012351", + "00012352", + "00012353", + "00012354", + "00012355", + "00012356", + "00012357", + "00012358", + "00012359", + "00012360", + ], + "claim_id": [ + "CLM-000123", + "CLM-000124", + "CLM-000125", + "CLM-000126", + "CLM-000127", + "CLM-000128", + "CLM-000129", + "CLM-000130", + "CLM-000131", + "CLM-000132", + "CLM-000133", + "CLM-000134", + "CLM-000135", + "CLM-000136", + "CLM-000137", + "CLM-000138", + ], + "premium": [1000.0] * 16, + "premium_currency": ["USD"] * 16, + "reserve": [ + 500.0, + 600.0, + 700.0, + 800.0, + -250.0, + 900.0, + 1000.0, + 1100.0, + 1200.0, + 1300.0, + 1400.0, + 1500.0, + 1600.0, + 1700.0, + 1800.0, + 1900.0, + ], + "reserve_currency": [ + "USD", + "USD", + "USD", + "EUR", + "USD", + "USD", + "USD", + "USD", + "USD", + "USD", + "USD", + "USD", + "USD", + "USD", + "USD", + "USD", + ], + "incident_date": ["2025-01-01"] * 16, + "report_date": ["2025-01-02"] * 16, + "state": [ + "open", + "closed", + "open", + "open", + "open", + "open", + "open", + "open", + "open", + "open", + "open", + "open", + "open", + "open", + "open", + "open", + ], + "claimant_name": ["Claimant"] * 16, + "loss_description": ["water damage"] * 16, + "batch": [f"seed-{int(seed)}"] * 16, + }, + index=rows, + ) + builder = FixtureBuilder( + "v1", + "insurance", + frame, + seed=int(seed), + schema={ + "columns": list(frame.columns), + "dtypes": {c: str(frame[c].dtype) for c in frame.columns}, + }, + policy={ + "reference_date": "2026-01-15", + "timezone": "UTC", + "locale": "en_US", + "currency": "USD", + "supported_currencies": ["USD", "EUR", "INR"], + "protected_policy_policy": "preserve", + }, + protected_columns=("policy_number",), + ) + builder.inject( + "ins-01", "policy_number", "00012345", Disposition.PRESERVE, family="policy-id-format" + ) + builder.inject( + "ins-02", "claim_id", "CLM-000123", Disposition.PRESERVE, family="claim-id-format" + ) + builder.inject( + "ins-03", + "premium", + "1,000.00", + Disposition.REPAIR, + expected=1000.0, + expected_dtype="float64", + family="grouped-premium", + ) + builder.inject( + "ins-04", + "reserve_currency", + "EUR", + Disposition.REVIEW, + family="premium-reserve-currency-conflict", + ) + builder.inject( + "ins-05", "reserve", -250.0, Disposition.REVIEW, family="negative-reserve-review" + ) + builder.inject( + "ins-06", + "report_date", + "2025-01-01", + Disposition.REVIEW, + family="incident-report-date-ordering", + ) + builder.inject( + "ins-07", + "state", + "open|closed", + Disposition.REVIEW, + family="state-transition-contradiction", + ) + builder.inject( + "ins-09", + "claimant_name", + "TB-INS-CLAIMANT-0001", + Disposition.FLAG, + family="claimant-pii", + sensitive=True, + ) + builder.inject( + "ins-10", + "loss_description", + "TB-INS-MEDICAL-0001", + Disposition.FLAG, + family="medical-loss-text", + sensitive=True, + ) + builder.inject( + "ins-16", + "policy_number", + "TB-INS-POLICY-TAIL", + Disposition.REVIEW, + family="protected-policy-number-conflict", + sensitive=True, + ) + builder.add_row_case( + "exact-duplicate-ins-02-ins-03", Disposition.FLAG, family="exact-duplicate" + ) + builder.add_row_case("removed-ins-15", Disposition.REVIEW, family="removed-row") + builder.add_schema_case("added-column", Disposition.REVIEW, family="added-column") + builder.add_schema_case("removed-column", Disposition.REVIEW, family="removed-column") + builder.add_schema_case("renamed-column", Disposition.REVIEW, family="renamed-column") + builder.add_schema_case("reordered-columns", Disposition.REVIEW, family="reordered-columns") + builder.add_schema_case( + "type-drifted-reserve", Disposition.REVIEW, family="type-drifted-column" + ) + return builder.build() diff --git a/benchmarks/truthbench/fixtures/logistics.py b/benchmarks/truthbench/fixtures/logistics.py new file mode 100644 index 0000000..db58c96 --- /dev/null +++ b/benchmarks/truthbench/fixtures/logistics.py @@ -0,0 +1,172 @@ +"""Deterministic logistics gold fixture for TruthBench.""" + +from __future__ import annotations + +import pandas as pd + +from ..models import Disposition +from .base import FixtureBuilder, TruthFixture + + +def build(seed: int = 1729) -> TruthFixture: + rows = [f"log-{i:02d}" for i in range(1, 17)] + frame = pd.DataFrame( + { + "shipment_id": [f"TB-LOG-{i:04d}" for i in range(1, 17)], + "origin_code": ["USLAX"] * 16, + "destination_code": [ + "NLRTM", + "DEHAM", + "GBFXT", + "INBOM", + "SGSIN", + "CNSHA", + "JPTYO", + "AUMEL", + "BRSSZ", + "ZADUR", + "AEJEA", + "NOOSL", + "ESBCN", + "CATOR", + "MYPKG", + "INBOM", + ], + "weight": [ + 10.0, + "10 lb", + 12.0, + 8.0, + 4.0, + 15.0, + 20.0, + 25.0, + 30.0, + 35.0, + 40.0, + 45.0, + 50.0, + 55.0, + 60.0, + 65.0, + ], + "weight_unit": ["kg"] * 16, + "temperature": [ + 20.0, + 21.0, + 98.6, + 19.0, + 18.0, + 22.0, + 23.0, + 24.0, + 25.0, + 26.0, + 27.0, + 28.0, + 29.0, + 30.0, + 31.0, + 32.0, + ], + "temperature_unit": ["C"] * 16, + "delivery_window": ["2026-01-15 09:00-17:00"] * 16, + "transport_time": ["12:00"] * 16, + "address": ["Warehouse"] * 16, + "tracking_status": ["on-time"] * 16, + "timezone": ["UTC"] * 16, + "batch": [f"seed-{int(seed)}"] * 16, + }, + index=rows, + ) + builder = FixtureBuilder( + "v1", + "logistics", + frame, + seed=int(seed), + schema={ + "columns": list(frame.columns), + "dtypes": {c: str(frame[c].dtype) for c in frame.columns}, + }, + policy={ + "reference_date": "2026-01-15", + "timezone": "UTC", + "locale": "en_US", + "weight_unit": "kg", + "temperature_unit": "C", + "protected_shipment_policy": "preserve", + }, + protected_columns=("shipment_id",), + ) + builder.inject( + "log-04", "destination_code", "INBOM", Disposition.PRESERVE, family="rare-unlocode-valid" + ) + builder.inject( + "log-02", + "weight", + "10 lb", + Disposition.REPAIR, + expected=4.5359237, + expected_dtype="float64", + family="kg-lb-conversion", + ) + builder.inject("log-02", "weight_unit", "lb", Disposition.PRESERVE, family="weight-unit-lb") + builder.inject( + "log-03", "temperature", 98.6, Disposition.REVIEW, family="celsius-fahrenheit-conflict" + ) + builder.inject( + "log-03", "temperature_unit", "F", Disposition.PRESERVE, family="temperature-unit-f" + ) + builder.inject( + "log-05", + "delivery_window", + "2026-01-15 23:30-2026-01-16 01:00", + Disposition.REVIEW, + family="cross-timezone-window", + ) + builder.inject( + "log-05", + "timezone", + "Asia/Kolkata→America/New_York", + Disposition.REVIEW, + family="cross-timezone-window", + ) + builder.inject( + "log-06", + "transport_time", + "24:00", + Disposition.REPAIR, + expected="00:00", + family="twentyfour-hour-transport", + ) + builder.inject( + "log-09", + "address", + "TB-LOG-ADDRESS-0001", + Disposition.FLAG, + family="address-pii", + sensitive=True, + ) + builder.inject( + "log-15", "tracking_status", "delayed", Disposition.FLAG, family="late-tracking-canary" + ) + builder.inject( + "log-16", + "shipment_id", + "TB-LOG-SHIPMENT-TAIL", + Disposition.REVIEW, + family="protected-shipment-id-conflict", + sensitive=True, + ) + builder.add_row_case( + "exact-duplicate-log-02-log-03", Disposition.FLAG, family="exact-duplicate" + ) + builder.add_row_case("removed-log-15", Disposition.REVIEW, family="removed-row") + builder.add_schema_case("added-column", Disposition.REVIEW, family="added-column") + builder.add_schema_case("removed-column", Disposition.REVIEW, family="removed-column") + builder.add_schema_case("renamed-column", Disposition.REVIEW, family="renamed-column") + builder.add_schema_case("reordered-columns", Disposition.REVIEW, family="reordered-columns") + builder.add_schema_case( + "type-drifted-weight", Disposition.REVIEW, family="type-drifted-column" + ) + return builder.build() diff --git a/benchmarks/truthbench/fixtures/retail.py b/benchmarks/truthbench/fixtures/retail.py new file mode 100644 index 0000000..be39bbf --- /dev/null +++ b/benchmarks/truthbench/fixtures/retail.py @@ -0,0 +1,194 @@ +"""Deterministic retail gold fixture for TruthBench.""" + +from __future__ import annotations + +import pandas as pd + +from ..models import Disposition +from .base import FixtureBuilder, TruthFixture + + +def build(seed: int = 1729) -> TruthFixture: + rows = [f"ret-{i:02d}" for i in range(1, 17)] + frame = pd.DataFrame( + { + "sku": [ + "001234", + "000007", + "123456", + "888888", + "100001", + "100002", + "100003", + "100004", + "100005", + "100006", + "100007", + "100008", + "100009", + "100010", + "100011", + "100012", + ], + "gtin": [ + "00012345678905", + "00000012345678", + "12345678901234", + "88888888888888", + "10000000000001", + "10000000000002", + "10000000000003", + "10000000000004", + "10000000000005", + "10000000000006", + "10000000000007", + "10000000000008", + "10000000000009", + "10000000000010", + "10000000000011", + "10000000000012", + ], + "product_name": [ + "Widget", + "Free Sample", + "Café & Tea", + "茶", + "Book", + "Lamp", + "Chair", + "Desk", + "Glass", + "Mug", + "Bag", + "Pencil", + "Shoes", + "Coat", + "Hat", + "Socks", + ], + "quantity": [1, 1, 2, 1, 1, -2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], + "price": [ + "12.50", + "0.00", + "1.234,56", + "₹1,23,456.70", + "9.99", + "10.00", + "11.00", + "12.00", + "13.00", + "14.00", + "15.00", + "16.00", + "17.00", + "18.00", + "19.00", + "20.00", + ], + "currency": [ + "USD", + "USD", + "EUR", + "INR", + "USD", + "USD", + "USD", + "USD", + "USD", + "USD", + "USD", + "USD", + "USD", + "USD", + "USD", + "USD", + ], + "review": ["great"] * 16, + "email": ["ordinary@example.invalid"] * 16, + "card": ["TB-RETAIL-CARD-0001"] * 16, + "order_date": ["2026-01-15"] * 16, + "batch": [f"seed-{int(seed)}"] * 16, + }, + index=rows, + ) + builder = FixtureBuilder( + "v1", + "retail", + frame, + seed=int(seed), + schema={ + "columns": list(frame.columns), + "dtypes": {c: str(frame[c].dtype) for c in frame.columns}, + }, + policy={ + "reference_date": "2026-01-15", + "timezone": "UTC", + "locale": "en_US", + "locales": ["en_US", "de_DE", "hi_IN", "zh_CN"], + "currency": "USD", + }, + protected_columns=("sku", "gtin"), + ) + builder.inject("ret-01", "sku", "001234", Disposition.PRESERVE, family="leading-zero-sku") + builder.inject( + "ret-01", "gtin", "00012345678905", Disposition.PRESERVE, family="leading-zero-gtin" + ) + builder.inject("ret-02", "price", "0.00", Disposition.PRESERVE, family="free-item") + builder.inject("ret-06", "quantity", -2, Disposition.FLAG, family="return-quantity") + builder.inject( + "ret-03", + "price", + "1.234,56", + Disposition.REPAIR, + expected=1234.56, + expected_dtype="float64", + family="mixed-decimal-grouping", + ) + builder.inject("ret-04", "price", "₹1,23,456.70", Disposition.REVIEW, family="mixed-currency") + builder.inject("ret-05", "currency", "EUR", Disposition.REVIEW, family="mixed-currency") + builder.inject( + "ret-03", + "product_name", + "Café & Tea", + Disposition.REVIEW, + family="html-entity-mojibake", + ) + builder.inject( + "ret-04", "product_name", "茶", Disposition.PRESERVE, family="multilingual-product" + ) + builder.inject( + "ret-07", + "review", + "customer@example.invalid", + Disposition.REVIEW, + family="email-in-review", + sensitive=True, + ) + builder.inject( + "ret-08", + "card", + "TB-RETAIL-CARD-REVIEW", + Disposition.REVIEW, + family="card-in-review", + sensitive=True, + ) + builder.inject( + "ret-16", + "email", + "tail@example.invalid", + Disposition.FLAG, + family="tail-email-canary", + sensitive=True, + ) + builder.add_row_case( + "exact-duplicate-ret-02-ret-03", Disposition.FLAG, family="exact-duplicate" + ) + builder.add_row_case("removed-ret-15", Disposition.REVIEW, family="removed-row") + builder.add_schema_case("added-column", Disposition.REVIEW, family="added-column") + builder.add_schema_case("removed-column", Disposition.REVIEW, family="removed-column") + builder.add_schema_case("renamed-column", Disposition.REVIEW, family="renamed-column") + builder.add_schema_case("reordered-columns", Disposition.REVIEW, family="reordered-columns") + builder.add_schema_case( + "type-drifted-quantity", Disposition.REVIEW, family="type-drifted-column" + ) + return builder.build() diff --git a/benchmarks/truthbench/gates.py b/benchmarks/truthbench/gates.py new file mode 100644 index 0000000..11fb518 --- /dev/null +++ b/benchmarks/truthbench/gates.py @@ -0,0 +1,454 @@ +"""Absolute, fail-closed TruthBench release gates.""" + +from __future__ import annotations + +import ast +import re +from collections.abc import Iterable, Mapping +from dataclasses import dataclass, field +from itertools import product +from types import MappingProxyType +from typing import Any + +from .exact import TypedValue, encode_typed +from .models import DecisionRecord, Disposition, GateResult, RunResult +from .privacy import SinkScanner + +_RAW_PII = re.compile(r"[A-Za-z0-9.!#$%&'*+/=?^_`{|}~-]+@[A-Za-z0-9.-]+\.invalid") +_BOILERPLATE = {"", "n/a", "none", "unknown", "automatic", "auto"} +_SURFACE_CLASSES = {"mutator", "validator", "pii", "explicit_transform"} + + +@dataclass(frozen=True) +class _FixtureEvidence: + """A copy-protected oracle snapshot, never a live fixture DataFrame.""" + + domain: str + cells: Mapping[str, Any] + inputs: Mapping[str, TypedValue] + protected_columns: frozenset[str] + canaries: tuple[tuple[str, Any], ...] + required_case_ids: frozenset[str] + + +def _snapshot(fixture: Any) -> _FixtureEvidence: + frame = fixture.frame + cells = {cell.cell_id: cell for cell in fixture.cells} + inputs = { + cell.cell_id: encode_typed( + frame.at[cell.row_id, cell.column], dtype=frame[cell.column].dtype + ) + for cell in fixture.cells + if not cell.sensitive + } + cases = (*getattr(fixture, "row_cases", ()), *getattr(fixture, "schema_cases", ())) + return _FixtureEvidence( + domain=str(fixture.domain), + cells=MappingProxyType(cells), + inputs=MappingProxyType(inputs), + protected_columns=frozenset(str(column) for column in fixture.protected_columns), + canaries=tuple((str(key), value) for key, value in fixture.pii_canaries.items()), + required_case_ids=frozenset(case.case_id for case in cases), + ) + + +@dataclass(frozen=True) +class GateRun: + """Immutable gate context paired with the existing serialized ``RunResult``. + + The public result schema remains untouched. At construction, live fixture + frames are converted into typed, mapping-proxy oracle snapshots so later + fixture mutation cannot change a release decision. + """ + + run: RunResult + fixtures: tuple[Any, ...] + generated_code: tuple[str, ...] = () + persisted_sinks: tuple[Any, ...] = () + complete: bool = True + schema_valid: bool = True + unexpected_exceptions: tuple[Any, ...] = () + expected_surfaces: tuple[str, ...] = () + expected_backends: tuple[str, ...] = () + expected_repeats: tuple[int, ...] = () + surface_classes: tuple[tuple[str, str], ...] = () + requested_cell_ids: tuple[str, ...] = () + case_records: tuple[Any, ...] = () + audit_record_ids: tuple[str, ...] = () + _evidence: tuple[_FixtureEvidence, ...] = field(init=False, repr=False) + + def __post_init__(self) -> None: + object.__setattr__( + self, "_evidence", tuple(_snapshot(fixture) for fixture in self.fixtures) + ) + + +@dataclass(frozen=True) +class GateEvaluation: + gates: tuple[GateResult, ...] + + @property + def passed(self) -> bool: + return all(gate.passed for gate in self.gates) + + +def _gate(name: str, failures: Iterable[str]) -> GateResult: + return GateResult(name, not (items := tuple(failures)), items) + + +def _cells(context: GateRun) -> dict[str, tuple[_FixtureEvidence, Any]]: + return { + cell_id: (evidence, cell) + for evidence in context._evidence + for cell_id, cell in evidence.cells.items() + } + + +def _surface_class(context: GateRun, surface: str) -> str | None: + return dict(context.surface_classes).get(surface) + + +def _is_mutator(context: GateRun, record: DecisionRecord) -> bool: + return _surface_class(context, record.surface) in {"mutator", "explicit_transform"} + + +def _changed(record: DecisionRecord) -> bool: + return record.actual_output is not None and record.actual_output != record.input + + +def _exact_repair(actual: TypedValue | None, expected: TypedValue | None) -> bool: + if actual is None or expected is None: + return False + if expected.dtype is not None: + return actual == expected + # Fixture authors who do not specify a dtype constrain scalar type/value, + # not an incidental object/string backend representation. + return ( + actual.kind == expected.kind + and actual.value == expected.value + and actual.redacted == expected.redacted + ) + + +def _substantive(record: DecisionRecord, audit_ids: set[str]) -> bool: + rationale = (record.rationale or "").strip() + return ( + rationale.casefold() not in _BOILERPLATE + and len(rationale) >= 12 + and bool((record.rule_id or "").strip()) + and record.audit_complete is True + and bool(set(record.audit_ids or ()) & audit_ids) + ) + + +def _canary_leaks(context: GateRun) -> tuple[str, ...]: + canaries = {key: value for evidence in context._evidence for key, value in evidence.canaries} + failures: list[str] = [] + if canaries: + scanner = SinkScanner.from_canaries(canaries, key=b"truthbench-gate-scan-v1") + for index, sink in enumerate(context.persisted_sinks): + for leak in scanner.scan(sink): + failures.append(f"sink[{index}] leaked canary {leak.canary_id} at {leak.path}") + for index, sink in enumerate(context.persisted_sinks): + if _RAW_PII.search(str(sink)): + failures.append(f"sink[{index}] contains raw synthetic PII") + return tuple(failures) + + +def _coverage_failures(context: GateRun, cells: Mapping[str, Any]) -> list[str]: + if ( + not context.expected_surfaces + or not context.expected_backends + or not context.expected_repeats + ): + return ["expected surface/backend/repeat coverage was not declared"] + failures: list[str] = [] + for cell_id, surface, backend, repeat in product( + cells, context.expected_surfaces, context.expected_backends, context.expected_repeats + ): + matches = [ + record + for record in context.run.records + if record.cell_id == cell_id + and record.surface == surface + and record.requested_backend == backend + and record.repeat == repeat + ] + if len(matches) != 1: + failures.append( + f"{cell_id}: expected exactly one {surface}/{backend}/repeat-{repeat} record" + ) + declared = dict(context.surface_classes) + for surface in context.expected_surfaces: + if declared.get(surface) not in _SURFACE_CLASSES: + failures.append(f"{surface}: surface class was not declared") + return failures + + +def evaluate_gates(context: GateRun) -> GateEvaluation: + """Evaluate all gates independently. No baseline or stale claim clears one.""" + + run, records, cells = context.run, context.run.records, _cells(context) + audit_ids = set(context.audit_record_ids) + gates: list[GateResult] = [] + coverage = _coverage_failures(context, cells) if cells else ["fixture policy/cells absent"] + gates.append( + _gate( + "completeness", + ["run is partial"] * (not context.complete or not records) + coverage, + ) + ) + gates.append( + _gate("schema_validation", ["schema validation failed"] * (not context.schema_valid)) + ) + gates.append( + _gate("fixture_evidence", ["fixture policy/cell evidence is absent"] * (not cells)) + ) + gates.append( + _gate( + "unexpected_exception", + (f"unexpected exception: {e}" for e in context.unexpected_exceptions), + ) + ) + required = set(context.expected_backends or run.required_backends) + actual = {record.actual_backend for record in records if record.actual_backend} + gates.append( + _gate( + "required_backend", + (f"required backend {b} was not executed" for b in sorted(required - actual)), + ) + ) + gates.append( + _gate( + "valid_value_corruption", + ( + f"{record.record_id}: preserve value was corrupted" + for record in records + if _is_mutator(context, record) + and record.expected_disposition is Disposition.PRESERVE + and _changed(record) + ), + ) + ) + gates.append( + _gate( + "protected_column_modification", + ( + f"{record.record_id}: protected column {record.column} was modified" + for record in records + if (item := cells.get(record.cell_id)) is not None + and record.column in item[0].protected_columns + and (record.mutated is True or _changed(record)) + ), + ) + ) + gates.append(_gate("raw_pii_leakage", _canary_leaks(context))) + gates.append( + _gate( + "backend_inconsistency", + ( + f"{record.record_id}: missing or inconsistent backend disclosure" + for record in records + if not record.requested_backend + or not record.actual_backend + or record.requested_backend != record.actual_backend + ), + ) + ) + gates.append( + _gate( + "default_nondeterminism", + ( + f"{record.record_id}: repeat decision is absent or inconsistent" + for record in records + if record.repeat_consistent is not True or not record.repeat_hash + ), + ) + ) + broken = [] + for index, code in enumerate(context.generated_code): + try: + ast.parse(code) + except SyntaxError: + broken.append(f"generated code[{index}] does not parse") + gates.append(_gate("broken_generated_code", broken)) + gates.append( + _gate( + "unexplained_high_confidence", + ( + f"{record.record_id}: high-confidence decision lacks rationale/provenance/audit" + for record in records + if record.confidence is not None + and record.confidence >= 0.90 + and not _substantive(record, audit_ids) + ), + ) + ) + gates.append( + _gate( + "trust_inversion", + ( + f"{record.record_id}: trust increased after data changed" + for record in records + if (record.mutated is True or _changed(record)) + and record.trust_before is not None + and record.trust_after is not None + and record.trust_after > record.trust_before + ), + ) + ) + gates.append( + _gate( + "mutation_audit", + ( + f"{record.record_id}: decision lacks matching audit evidence" + for record in records + if ( + record.mutated + or record.detected + or (_is_mutator(context, record) and _changed(record)) + ) + and ( + record.audit_complete is not True + or not set(record.audit_ids or ()) & audit_ids + ) + ), + ) + ) + gates.append( + _gate( + "review_routing", + ( + f"{record.record_id}: review case was not routed to a human" + for record in records + if record.expected_disposition is Disposition.REVIEW + and record.human_review is not True + ), + ) + ) + gates.append( + _gate( + "exact_repair", + ( + f"{record.record_id}: repair differs from the exact oracle" + for record in records + if _is_mutator(context, record) + and record.expected_disposition is Disposition.REPAIR + and not _exact_repair(record.actual_output, record.expected_output) + ), + ) + ) + gates.append( + _gate( + "flag_mutation", + ( + f"{record.record_id}: flag/review value was mutated" + for record in records + if _is_mutator(context, record) + and record.expected_disposition in {Disposition.FLAG, Disposition.REVIEW} + and (record.mutated is True or _changed(record)) + ), + ) + ) + gates.append( + _gate( + "validator_contract", + ( + f"{record.record_id}: validator mutated data or missed detection" + for record in records + if _surface_class(context, record.surface) == "validator" + and ( + record.mutated is True + or _changed(record) + or ( + record.expected_disposition is not Disposition.PRESERVE + and record.detected is not True + ) + ) + ), + ) + ) + gates.append( + _gate( + "pii_scope", + ( + f"{record.record_id}: PII surface missed a canary or changed a false positive" + for record in records + if _surface_class(context, record.surface) == "pii" + and ( + (record.sensitive and record.detected is not True) + or ( + not record.sensitive + and (record.mutated is True or _changed(record) or record.detected is True) + ) + ) + ), + ) + ) + requested = set(context.requested_cell_ids) + gates.append( + _gate( + "requested_behavior", + ( + ( + f"{record.record_id}: explicit transform changed an unrequested " + "cell or missed requested behavior" + ) + for record in records + if _surface_class(context, record.surface) == "explicit_transform" + and ( + ( + record.cell_id not in requested + and (record.mutated is True or _changed(record)) + ) + or ( + record.cell_id in requested + and record.expected_disposition is Disposition.REPAIR + and not _exact_repair(record.actual_output, record.expected_output) + ) + ) + ), + ) + ) + gates.append( + _gate( + "input_mutation", + ( + f"{record.record_id}: input evidence differs from fixture snapshot" + for record in records + if (item := cells.get(record.cell_id)) is not None + and not record.sensitive + and record.input != item[0].inputs[record.cell_id] + ), + ) + ) + required_cases = { + case for evidence in context._evidence for case in evidence.required_case_ids + } + observed_cases = { + case.case_id for case in context.case_records if getattr(case, "observed", False) + } + gates.append( + _gate( + "case_coverage", + ( + f"required case {case} was not observed" + for case in sorted(required_cases - observed_cases) + ), + ) + ) + gates.append( + _gate( + "aggregate_consistency", + ["summary record aggregate does not match records"] + * (dict(run.summary).get("records") != len(records)), + ) + ) + return GateEvaluation(tuple(gates)) + + +def failed_gate_names(result: GateEvaluation) -> set[str]: + return {gate.name for gate in result.gates if not gate.passed} + + +__all__ = ["GateEvaluation", "GateRun", "evaluate_gates", "failed_gate_names"] diff --git a/benchmarks/truthbench/inventory.py b/benchmarks/truthbench/inventory.py new file mode 100644 index 0000000..1b58ea8 --- /dev/null +++ b/benchmarks/truthbench/inventory.py @@ -0,0 +1,348 @@ +"""Manifest of FreshData's public decision, sink, and caller-directed surfaces.""" + +from __future__ import annotations + +import argparse +from collections.abc import Iterable +from dataclasses import dataclass +from enum import Enum + +import freshdata as fd +from freshdata.domains.registry import available as available_domains + + +class SurfaceClass(str, Enum): + DECISION = "decision" + SINK = "sink" + EXPLICIT_TRANSFORM = "explicit-transform" + DATA_MODEL = "data-model" + CONFIGURATION = "configuration" + REGISTRATION = "registration" + OUT_OF_SCOPE = "out-of-scope-with-reason" + + +@dataclass(frozen=True) +class SurfaceSpec: + name: str + version: int + classification: SurfaceClass + adapter: str | None + mutates: bool + deterministic: bool + backend_parity: bool + rationale: str + + def __post_init__(self) -> None: + if not self.name: + raise ValueError("surface name must not be empty") + if self.version != 1: + raise ValueError("unsupported surface spec version") + try: + classification = SurfaceClass(self.classification) + except ValueError as exc: + raise ValueError(f"unknown surface classification: {self.classification!r}") from exc + object.__setattr__(self, "classification", classification) + if classification in {SurfaceClass.DECISION, SurfaceClass.SINK} and not self.adapter: + raise ValueError(f"{classification.value} surface {self.name!r} requires an adapter") + if not self.rationale: + raise ValueError(f"surface {self.name!r} requires a rationale") + + +_CALLER_DIRECTED = { + "fill_missing", + "remove_outliers", + "resolve_duplicates", + "group_aggregate", + "set_display", + "get_display", + "reset_display", + "tokenize_value", + "detokenize_value", + "detokenize_series", + "make_vault", + "vault_metadata", + "anonymize", + "clean_text", + "clean_text_value", + "lint_text_encoding", + "mask_dataframe", +} +_REGISTRATION = { + "register_backend", + "register_comparator", + "register_expert", + "register_exporter", + "register_validator", + "registered_plugins", + "available_packs", +} +_SINKS = { + "build_exception_table", + "build_quality_report", + "export", + "export_dbt_tests", + "export_gx_suite", + "export_quality_ops", + "export_review_queue", + "generate_compliance_report", + "insight_report", + "stakeholder_summary", + "trust_gate_report", + "save_baseline", +} +_DECISIONS = { + "apply", + "apply_field_policy", + "apply_plan", + "apply_privacy_policy", + "apply_review_decisions", + "build_review_queue", + "cdc_profile", + "clean", + "clean_csv", + "clean_domain_file", + "clean_enterprise", + "clean_timeseries", + "cluster_column", + "compare_clean", + "compare_plans", + "compile_context", + "compute_trust_score", + "detect_outliers", + "detect_pii", + "diff_schema", + "enforce_contract", + "evaluate_quality_debt", + "explain_clean", + "infer_roles", + "learn", + "learn_cleaning_memory", + "link", + "link_entities", + "monitor_contract", + "parse_domain", + "pipeline", + "pipeline_clean", + "plan", + "profile", + "recalibrate_weights", + "redaction_columns", + "resolve_entities", + "run_semantic_validation", + "run_suite", + "suggest_join_keys", + "suggest_plan", + "validate", + "validate_fields", + "analyze_dataset", + "check_k_anonymity", + "classify_columns", + "compare_to_baseline", + "merge_clusters", + "merge_entities", + "schema_of", + "load_review_decisions", +} +_CONFIG_FUNCTIONS = { + "education_template", + "build_baseline", + "get_template", + "healthcare_template", + "load_baseline", + "load_cleaning_memory", + "load_compliance_pack", + "load_privacy_policy", + "load_profile", + "media_template", + "retail_template", + "save_profile", +} +_OUT_OF_SCOPE = {"enterprise", "models", "testing", "__version__"} + + +def cli_commands() -> tuple[str, ...]: + """Return top-level and nested enterprise CLI command paths.""" + + from freshdata.enterprise.cli import build_parser + + parser = build_parser() + result: set[str] = set() + + def visit(current: argparse.ArgumentParser, prefix: str = "") -> None: + for action in current._actions: + if not isinstance(action, argparse._SubParsersAction): + continue + choices = action.choices + for command, child in choices.items(): + path = f"{prefix} {command}".strip() + result.add(path) + if isinstance(child, argparse.ArgumentParser): + visit(child, path) + + visit(parser) + return tuple(sorted(result)) + + +def discover_public_names() -> tuple[str, ...]: + """Discover all names and registered command/domain surfaces in this checkout.""" + + names = set(fd.__dir__()) + from freshdata.experimental import ai_copilot + + names.update(ai_copilot.__all__) + names.update(f"domain:{name}" for name in available_domains()) + names.update(f"cli:{name}" for name in cli_commands()) + return tuple(sorted(names)) + + +def _classify(name: str) -> tuple[SurfaceClass, str | None, bool, bool, bool, str]: + if name.startswith("domain:"): + return ( + SurfaceClass.DECISION, + "generic", + False, + True, + True, + "bundled domain validator decision surface", + ) + if name.startswith("cli:"): + return ( + SurfaceClass.DECISION, + "generic", + False, + True, + False, + "enterprise CLI decision/report command", + ) + if name in _CALLER_DIRECTED: + return ( + SurfaceClass.EXPLICIT_TRANSFORM, + None, + name not in {"get_display", "reset_display", "vault_metadata"}, + True, + False, + "caller-directed explicit transform; no inferred disposition credit", + ) + if name in _REGISTRATION: + return ( + SurfaceClass.REGISTRATION, + None, + True, + True, + False, + "plugin registration or registry inspection", + ) + if name in _CONFIG_FUNCTIONS: + return ( + SurfaceClass.CONFIGURATION, + None, + False, + True, + False, + "configuration/template loading surface", + ) + if name in _SINKS: + return ( + SurfaceClass.SINK, + "generic", + False, + True, + False, + "report, audit, or export sink", + ) + if name in _DECISIONS: + return ( + SurfaceClass.DECISION, + "generic", + name.startswith(("clean", "apply", "run_")), + True, + True, + "public FreshData decision surface", + ) + if name in _OUT_OF_SCOPE: + return ( + SurfaceClass.OUT_OF_SCOPE, + None, + False, + True, + False, + "namespace or metadata surface; no cell disposition", + ) + if name.startswith("__"): + return SurfaceClass.CONFIGURATION, None, False, True, False, "package metadata" + if name.endswith(("Config", "Policy", "Rule", "Spec", "Weights")) or name in { + "Action", + "Jurisdiction", + "CompliancePack", + }: + return ( + SurfaceClass.CONFIGURATION, + None, + False, + True, + False, + "configuration or policy model", + ) + if name[:1].isupper(): + return SurfaceClass.DATA_MODEL, None, False, True, False, "public data/report model" + raise ValueError(f"unclassified FreshData public surface: {name}") + + +def build_manifest(names: Iterable[str] | None = None) -> tuple[SurfaceSpec, ...]: + """Build an immutable manifest from the currently discovered public surfaces.""" + + selected = discover_public_names() if names is None else tuple(sorted(set(names))) + specs: list[SurfaceSpec] = [] + for name in selected: + classification, adapter, mutates, deterministic, parity, rationale = _classify(name) + specs.append( + SurfaceSpec( + name=name, + version=1, + classification=classification, + adapter=adapter, + mutates=mutates, + deterministic=deterministic, + backend_parity=parity, + rationale=rationale, + ) + ) + validate_manifest(specs) + return tuple(specs) + + +def validate_manifest(manifest: Iterable[SurfaceSpec]) -> None: + """Validate uniqueness, classifications, and adapter requirements.""" + + specs = tuple(manifest) + names = [spec.name for spec in specs] + if len(names) != len(set(names)): + raise ValueError("surface manifest contains duplicate names") + from .surfaces import adapters + + registered = adapters() + for spec in specs: + if not isinstance(spec, SurfaceSpec): + raise TypeError("surface manifest entries must be SurfaceSpec instances") + # Re-run constructor checks for objects supplied by callers or tests. + SurfaceSpec(**{field: getattr(spec, field) for field in SurfaceSpec.__dataclass_fields__}) + if ( + spec.classification in {SurfaceClass.DECISION, SurfaceClass.SINK} + and spec.adapter not in registered + ): + raise ValueError(f"surface {spec.name!r} references unknown adapter {spec.adapter!r}") + + +SURFACE_MANIFEST = build_manifest() +PUBLIC_SURFACES = SURFACE_MANIFEST + +__all__ = [ + "PUBLIC_SURFACES", + "SURFACE_MANIFEST", + "SurfaceClass", + "SurfaceSpec", + "build_manifest", + "cli_commands", + "discover_public_names", + "validate_manifest", +] diff --git a/benchmarks/truthbench/models.py b/benchmarks/truthbench/models.py new file mode 100644 index 0000000..2815667 --- /dev/null +++ b/benchmarks/truthbench/models.py @@ -0,0 +1,362 @@ +from __future__ import annotations + +from dataclasses import dataclass, field +from enum import Enum +from typing import Any + +from .exact import JsonValue, TypedValue, encode_typed + + +class Disposition(str, Enum): + PRESERVE = "preserve" + REPAIR = "repair" + FLAG = "flag" + REVIEW = "review" + + +class _Unset: + pass + + +UNSET = _Unset() +_TEST_DIGEST_KEY = b"truthbench-decision-record-for-test-v1" + + +def _disposition(value: Disposition | str) -> Disposition: + try: + return Disposition(value) + except ValueError as exc: + raise ValueError(f"unknown TruthBench disposition: {value!r}") from exc + + +def _stable_id(*components: str) -> str: + if any(not isinstance(item, str) or not item for item in components): + raise ValueError("stable ID components must be non-empty strings") + if any(":" in item for item in components): + raise ValueError("stable ID components must not contain a colon") + return ":".join(components) + + +@dataclass(frozen=True) +class GoldCell: + cell_id: str + fixture_version: str + domain: str + row_id: str + column: str + disposition: Disposition + expected_output: TypedValue | None = None + family: str | None = None + sensitive: bool = False + canary_id: str | None = None + schema_version: int = field(default=1, init=False) + + @classmethod + def create( + cls, + fixture_version: str, + domain: str, + row_id: str, + column: str, + disposition: Disposition | str, + *, + expected_output: Any = UNSET, + expected_dtype: Any = None, + family: str | None = None, + sensitive: bool = False, + canary_id: str | None = None, + digest_key: bytes = _TEST_DIGEST_KEY, + ) -> GoldCell: + cell_id = _stable_id(fixture_version, domain, row_id, column) + typed_output = None + if expected_output is not UNSET: + typed_output = encode_typed( + expected_output, + dtype=expected_dtype, + sensitive=sensitive, + digest_key=digest_key if sensitive else None, + ) + return cls( + cell_id=cell_id, + fixture_version=fixture_version, + domain=domain, + row_id=row_id, + column=column, + disposition=_disposition(disposition), + expected_output=typed_output, + family=family, + sensitive=sensitive, + canary_id=canary_id, + ) + + def to_dict(self) -> dict[str, JsonValue]: + return { + "schema_version": self.schema_version, + "cell_id": self.cell_id, + "fixture_version": self.fixture_version, + "domain": self.domain, + "row_id": self.row_id, + "column": self.column, + "disposition": self.disposition.value, + "expected_output": ( + None if self.expected_output is None else self.expected_output.to_dict() + ), + "family": self.family, + "sensitive": self.sensitive, + "canary_id": self.canary_id, + } + + +@dataclass(frozen=True) +class CaseExpectation: + case_id: str + fixture_version: str + domain: str + kind: str + name: str + disposition: Disposition + expected: TypedValue | None = None + family: str | None = None + sensitive: bool = False + schema_version: int = field(default=1, init=False) + + @classmethod + def create( + cls, + fixture_version: str, + domain: str, + kind: str, + name: str, + disposition: Disposition | str, + *, + expected: Any = UNSET, + expected_dtype: Any = None, + family: str | None = None, + sensitive: bool = False, + digest_key: bytes = _TEST_DIGEST_KEY, + ) -> CaseExpectation: + case_id = _stable_id(fixture_version, domain, kind, name) + typed_expected = None + if expected is not UNSET: + typed_expected = encode_typed( + expected, + dtype=expected_dtype, + sensitive=sensitive, + digest_key=digest_key if sensitive else None, + ) + return cls( + case_id=case_id, + fixture_version=fixture_version, + domain=domain, + kind=kind, + name=name, + disposition=_disposition(disposition), + expected=typed_expected, + family=family, + sensitive=sensitive, + ) + + def to_dict(self) -> dict[str, JsonValue]: + return { + "schema_version": self.schema_version, + "case_id": self.case_id, + "fixture_version": self.fixture_version, + "domain": self.domain, + "kind": self.kind, + "name": self.name, + "disposition": self.disposition.value, + "expected": None if self.expected is None else self.expected.to_dict(), + "family": self.family, + "sensitive": self.sensitive, + } + + +@dataclass(frozen=True) +class DecisionRecord: + record_id: str + run_id: str + fixture_id: str + case_id: str | None + cell_id: str + domain: str + row_id: str + column: str + surface: str + repeat: int + expected_disposition: Disposition + actual_disposition: Disposition | None + sensitive: bool + input: TypedValue + expected_output: TypedValue | None + actual_output: TypedValue | None + confidence: float | None = None + risk: str | None = None + status: str | None = None + rule_id: str | None = None + rationale: str | None = None + evidence_kinds: tuple[str, ...] | None = None + mutated: bool | None = None + detected: bool | None = None + quarantined: bool | None = None + human_review: bool | None = None + audit_required: bool | None = None + audit_complete: bool | None = None + audit_ids: tuple[str, ...] | None = None + trust_before: float | None = None + trust_after: float | None = None + trust_delta: float | None = None + requested_backend: str | None = None + actual_backend: str | None = None + fallback_events: tuple[str, ...] | None = None + backend_differences: tuple[str, ...] | None = None + normalized_decision_hash: str | None = None + repeat_hash: str | None = None + repeat_consistent: bool | None = None + schema_version: int = field(default=1, init=False) + + def __post_init__(self) -> None: + if not self.sensitive: + return + for name in ("input", "expected_output", "actual_output"): + value = getattr(self, name) + if value is not None and not ( + value.redacted + and value.value is None + and value.display == "[REDACTED]" + and bool(value.digest) + ): + raise ValueError( + f"sensitive DecisionRecord {name} must be a redacted TypedValue" + ) + + @classmethod + def for_test(cls, *, cell: GoldCell, input_value: Any) -> DecisionRecord: + typed_input = encode_typed( + input_value, + sensitive=cell.sensitive, + digest_key=_TEST_DIGEST_KEY if cell.sensitive else None, + ) + return cls( + record_id=f"test:{cell.cell_id}", + run_id="test", + fixture_id=f"{cell.fixture_version}:{cell.domain}", + case_id=None, + cell_id=cell.cell_id, + domain=cell.domain, + row_id=cell.row_id, + column=cell.column, + surface="test", + repeat=0, + expected_disposition=cell.disposition, + actual_disposition=None, + input=typed_input, + expected_output=cell.expected_output, + actual_output=None, + sensitive=cell.sensitive, + ) + + @staticmethod + def _typed(value: TypedValue | None) -> dict[str, JsonValue] | None: + return None if value is None else value.to_dict() + + def to_dict(self) -> dict[str, JsonValue]: + return { + "schema_version": self.schema_version, + "record_id": self.record_id, + "run_id": self.run_id, + "fixture_id": self.fixture_id, + "case_id": self.case_id, + "cell_id": self.cell_id, + "domain": self.domain, + "row_id": self.row_id, + "column": self.column, + "surface": self.surface, + "repeat": self.repeat, + "expected_disposition": self.expected_disposition.value, + "actual_disposition": ( + None if self.actual_disposition is None else self.actual_disposition.value + ), + "sensitive": self.sensitive, + "input": self.input.to_dict(), + "input_type": self.input.type_label, + "expected_output": self._typed(self.expected_output), + "expected_output_type": ( + None if self.expected_output is None else self.expected_output.type_label + ), + "actual_output": self._typed(self.actual_output), + "actual_output_type": ( + None if self.actual_output is None else self.actual_output.type_label + ), + "confidence": self.confidence, + "risk": self.risk, + "status": self.status, + "rule_id": self.rule_id, + "rationale": self.rationale, + "evidence_kinds": ( + None if self.evidence_kinds is None else list(self.evidence_kinds) + ), + "mutated": self.mutated, + "detected": self.detected, + "quarantined": self.quarantined, + "human_review": self.human_review, + "audit_required": self.audit_required, + "audit_complete": self.audit_complete, + "audit_ids": None if self.audit_ids is None else list(self.audit_ids), + "trust_before": self.trust_before, + "trust_after": self.trust_after, + "trust_delta": self.trust_delta, + "requested_backend": self.requested_backend, + "actual_backend": self.actual_backend, + "fallback_events": ( + None if self.fallback_events is None else list(self.fallback_events) + ), + "backend_differences": ( + None if self.backend_differences is None else list(self.backend_differences) + ), + "normalized_decision_hash": self.normalized_decision_hash, + "repeat_hash": self.repeat_hash, + "repeat_consistent": self.repeat_consistent, + } + + +@dataclass(frozen=True) +class GateResult: + name: str + passed: bool + failures: tuple[str, ...] = () + schema_version: int = field(default=1, init=False) + + def to_dict(self) -> dict[str, JsonValue]: + return { + "schema_version": self.schema_version, + "name": self.name, + "passed": self.passed, + "failure_count": len(self.failures), + "failures": list(self.failures), + } + + +@dataclass(frozen=True) +class RunResult: + run_id: str + profile: str + fixture_hashes: tuple[tuple[str, str], ...] + required_backends: tuple[str, ...] + records: tuple[DecisionRecord, ...] + gates: tuple[GateResult, ...] + summary: tuple[tuple[str, JsonValue], ...] + environment: tuple[tuple[str, JsonValue], ...] + schema_version: int = field(default=1, init=False) + + def to_dict(self) -> dict[str, JsonValue]: + return { + "schema_version": self.schema_version, + "run_id": self.run_id, + "profile": self.profile, + "fixture_hashes": dict(self.fixture_hashes), + "required_backends": list(self.required_backends), + "records": [record.to_dict() for record in self.records], + "gates": [gate.to_dict() for gate in self.gates], + "summary": dict(self.summary), + "environment": dict(self.environment), + } diff --git a/benchmarks/truthbench/normalize.py b/benchmarks/truthbench/normalize.py new file mode 100644 index 0000000..afd3b61 --- /dev/null +++ b/benchmarks/truthbench/normalize.py @@ -0,0 +1,274 @@ +"""Normalize heterogeneous public-surface evidence into TruthBench records. + +This module deliberately has no cleaning policy. It only captures what a +public surface did, preserves typed values, and leaves pass/fail decisions to +``gates``. Missing evidence therefore remains missing rather than being +silently inferred as a successful repair. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass +from typing import Any + +import pandas as pd + +from .exact import encode_typed, exact_equal +from .models import DecisionRecord, Disposition +from .surfaces.base import SurfaceObservation + +_SENSITIVE_KEY = b"truthbench-normalized-sensitive-v1" + + +@dataclass(frozen=True) +class CaseRecord: + """Non-serialized row/schema evidence retained for gate context. + + ``RunResult`` intentionally serializes only ``DecisionRecord``. Cases are + contextual evidence for the runner and release gates, never an alternative + public result schema. + """ + + case_id: str + kind: str + expected_disposition: Disposition + observed: bool + + +@dataclass(frozen=True) +class NormalizationResult: + records: tuple[DecisionRecord, ...] + cases: tuple[CaseRecord, ...] + input_mutated: bool = False + + +def _mapping(value: Any) -> Mapping[str, Any]: + return value if isinstance(value, Mapping) else {} + + +def _decision_for(raw: Any, cell_id: str) -> Mapping[str, Any]: + """Extract one cell's public decision without assuming a report type.""" + + source = _mapping(raw) + direct = source.get(cell_id) + if isinstance(direct, Mapping): + return direct + for name in ("decisions", "records", "actions", "findings"): + values = source.get(name) + if isinstance(values, Mapping) and isinstance(values.get(cell_id), Mapping): + return values[cell_id] + if isinstance(values, (list, tuple)): + for item in values: + if not isinstance(item, Mapping): + continue + if item.get("cell_id") == cell_id or item.get("id") == cell_id: + return item + return {} + + +def _audit_ids(sinks: Any, cell_id: str) -> tuple[str, ...]: + """Collect matching IDs from public audit payloads, never their contents.""" + + found: list[str] = [] + + def visit(value: Any) -> None: + if isinstance(value, Mapping): + for key, item in value.items(): + if key in {"record_id", "cell_id", "id"} and item == cell_id: + found.append(cell_id) + if ( + key in {"record_ids", "cell_ids", "ids"} + and isinstance(item, (list, tuple)) + and cell_id in item + ): + found.append(cell_id) + visit(item) + elif isinstance(value, (list, tuple)): + for item in value: + visit(item) + + visit(sinks) + return (cell_id,) if found else () + + +def _bool(decision: Mapping[str, Any], name: str, default: bool | None = None) -> bool | None: + value = decision.get(name, default) + return value if isinstance(value, bool) else default + + +def _disposition(value: Any) -> Disposition | None: + if value is None: + return None + try: + return Disposition(value) + except ValueError: + return None + + +def _output_value( + fixture: Any, cell: Any, observation: SurfaceObservation +) -> tuple[Any, Any] | None: + frame = observation.output_frame + if not isinstance(frame, pd.DataFrame): + return None + if cell.row_id not in frame.index or cell.column not in frame.columns: + return None + return frame.at[cell.row_id, cell.column], frame[cell.column].dtype + + +def _case_observed(raw: Any, case_id: str) -> bool: + """Return explicit row/schema observation evidence without inventing it.""" + + decision = _decision_for(raw, case_id) + return decision.get("observed") is True or decision.get("detected") is True + + +def normalize_observation( + fixture: Any, + observation: SurfaceObservation, + *, + surface: str, + backend: str, + repeat: int, + run_id: str, +) -> NormalizationResult: + """Produce one record for every labelled cell in ``fixture``. + + Validators may detect bad values without returning a frame. In that case + ``actual_output`` remains ``None`` and ``mutated`` remains false; gates can + give detection credit without inventing a mutation. + """ + + frame = getattr(fixture, "frame", None) + if not isinstance(frame, pd.DataFrame): + raise TypeError("normalization requires a pandas TruthFixture frame") + raw = observation.raw_decisions + trust = _mapping(observation.trust) + disclosure = _mapping(observation.backend_disclosure) + records: list[DecisionRecord] = [] + for cell in fixture.cells: + decision = _decision_for(raw, cell.cell_id) + source_value = frame.at[cell.row_id, cell.column] + source_dtype = frame[cell.column].dtype + output = _output_value(fixture, cell, observation) + actual_value, actual_dtype = output if output is not None else (None, None) + changed = output is not None and not exact_equal( + actual_value, source_value, left_dtype=actual_dtype, right_dtype=source_dtype + ) + explicit_mutated = _bool(decision, "mutated") + mutated = explicit_mutated if explicit_mutated is not None else changed + detected = _bool(decision, "detected", False) + quarantined = _bool(decision, "quarantined", False) + human_review = _bool(decision, "human_review", False) + actual_disposition = _disposition(decision.get("disposition")) + if actual_disposition is None: + if cell.disposition is Disposition.FLAG and (detected or quarantined): + actual_disposition = Disposition.FLAG + elif cell.disposition is Disposition.REVIEW and (detected or human_review): + actual_disposition = Disposition.REVIEW + elif cell.disposition is Disposition.REPAIR and (mutated or output is not None): + actual_disposition = Disposition.REPAIR + elif not mutated: + actual_disposition = Disposition.PRESERVE + audit_ids = _audit_ids(observation.audit_sinks, cell.cell_id) + audit_required = ( + cell.disposition is not Disposition.PRESERVE or bool(mutated) or bool(detected) + ) + confidence = decision.get("confidence") + if not isinstance(confidence, (int, float)) or isinstance(confidence, bool): + confidence = None + trust_before = trust.get("trust_before") + trust_after = trust.get("trust_after") + if not isinstance(trust_before, (int, float)) or isinstance(trust_before, bool): + trust_before = None + if not isinstance(trust_after, (int, float)) or isinstance(trust_after, bool): + trust_after = None + records.append( + DecisionRecord( + record_id=f"{run_id}:{surface}:{backend}:{repeat}:{cell.cell_id}", + run_id=run_id, + fixture_id=f"{cell.fixture_version}:{cell.domain}", + case_id=None, + cell_id=cell.cell_id, + domain=cell.domain, + row_id=cell.row_id, + column=cell.column, + surface=surface, + repeat=repeat, + expected_disposition=cell.disposition, + actual_disposition=actual_disposition, + sensitive=cell.sensitive, + input=encode_typed( + source_value, + dtype=source_dtype, + sensitive=cell.sensitive, + digest_key=_SENSITIVE_KEY if cell.sensitive else None, + ), + expected_output=cell.expected_output, + actual_output=( + None + if output is None + else encode_typed( + actual_value, + dtype=actual_dtype, + sensitive=cell.sensitive, + digest_key=_SENSITIVE_KEY if cell.sensitive else None, + ) + ), + confidence=float(confidence) if confidence is not None else None, + risk=decision.get("risk") if isinstance(decision.get("risk"), str) else None, + status=decision.get("status") if isinstance(decision.get("status"), str) else None, + rule_id=( + decision.get("rule_id") + if isinstance(decision.get("rule_id"), str) + else decision.get("model_id") + if isinstance(decision.get("model_id"), str) + else None + ), + rationale=decision.get("rationale") + if isinstance(decision.get("rationale"), str) + else None, + evidence_kinds=tuple( + str(item) + for item in decision.get("evidence_kinds", ()) + if isinstance(item, str) + ) + or None, + mutated=mutated, + detected=detected, + quarantined=quarantined, + human_review=human_review, + audit_required=audit_required, + audit_complete=(bool(audit_ids) if audit_required else True), + audit_ids=audit_ids or None, + trust_before=float(trust_before) if trust_before is not None else None, + trust_after=float(trust_after) if trust_after is not None else None, + trust_delta=( + round(float(trust_after - trust_before), 12) + if trust_before is not None and trust_after is not None + else None + ), + requested_backend=( + disclosure.get("requested") + if isinstance(disclosure.get("requested"), str) + else None + ), + actual_backend=( + disclosure.get("actual") if isinstance(disclosure.get("actual"), str) else None + ), + ) + ) + cases = tuple( + CaseRecord( + case.case_id, + case.kind, + case.disposition, + _case_observed(raw, case.case_id), + ) + for case in (*getattr(fixture, "row_cases", ()), *getattr(fixture, "schema_cases", ())) + ) + return NormalizationResult(tuple(records), cases) + + +__all__ = ["CaseRecord", "NormalizationResult", "normalize_observation"] diff --git a/benchmarks/truthbench/privacy.py b/benchmarks/truthbench/privacy.py new file mode 100644 index 0000000..052579d --- /dev/null +++ b/benchmarks/truthbench/privacy.py @@ -0,0 +1,680 @@ +"""Privacy-safe TruthBench sink values and exhaustive canary scanning. + +The scanner deliberately stores only canary identifiers, transform names, paths, +and run-scoped HMAC digests. Matched input is never included in a ``Leak`` or +in a redaction marker. +""" + +from __future__ import annotations + +import dataclasses +import hashlib +import hmac +import html +import json +import re +import unicodedata +import urllib.parse +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from typing import Any, Callable + +import pandas as pd + +from .exact import canonical_json, encode_typed + + +@dataclass(frozen=True) +class Leak: + """A privacy leak without retaining the matched value.""" + + canary_id: str + variant: str + path: str + + +@dataclass(frozen=True) +class PrivacySafeValue: + """A leaf value represented by a redacted display and keyed digest.""" + + value: str = "[REDACTED]" + digest: str = "" + + def __repr__(self) -> str: + return f"PrivacySafeValue(value={self.value!r}, digest={self.digest!r})" + + +_ZERO_WIDTH = "\u200b\u200c\u200d\ufeff" + + +def _remove_zero_width(value: str) -> str: + return "".join( + char for char in value if char not in _ZERO_WIDTH and unicodedata.category(char) != "Cf" + ) + + +def _remove_punctuation(value: str) -> str: + return "".join(char for char in value if not unicodedata.category(char).startswith("P")) + + +def _json_escaped(value: str) -> str: + encoded = json.dumps(value, ensure_ascii=True) + return encoded[1:-1] + + +def _digit_only(value: str) -> str: + return "".join(char for char in value if char.isdigit()) + + +Transform = Callable[[str], str] + +# Public names make the normalization contract inspectable and testable. +NORMALIZERS: tuple[tuple[str, Transform], ...] = ( + ("literal", lambda value: value), + ("casefold", str.casefold), + ("whitespace-stripped", lambda value: re.sub(r"\s+", "", value)), + ("punctuation-stripped", _remove_punctuation), + ("digit-only", _digit_only), + ("url-decoded", urllib.parse.unquote), + ("html-unescaped", html.unescape), + ("nfkc", lambda value: unicodedata.normalize("NFKC", value)), + ("nfc", lambda value: unicodedata.normalize("NFC", value)), + ("nfd", lambda value: unicodedata.normalize("NFD", value)), + ("zero-width-removed", _remove_zero_width), + ("json-escaped", _json_escaped), +) + + +def normalize_variants(value: str) -> dict[str, str]: + """Return named normalized forms of text, omitting empty forms.""" + + if not isinstance(value, str): + raise TypeError("normalize_variants expects text") + result: dict[str, str] = {} + for name, transform in NORMALIZERS: + try: + normalized = transform(value) + except (UnicodeError, ValueError): + continue + if normalized: + result.setdefault(name, normalized) + return result + + +def _text_forms(value: Any) -> dict[str, str]: + """Convert text/bytes to forms which can be safely searched.""" + + if isinstance(value, str): + return normalize_variants(value) + if isinstance(value, bytes): + forms: dict[str, str] = {} + try: + decoded = value.decode("utf-8") + except UnicodeDecodeError: + decoded = "" + if decoded: + for name, text in normalize_variants(decoded).items(): + forms.setdefault(name, text) + forms["bytes-hex"] = value.hex() + forms["bytes-escape"] = value.decode("latin1") + forms["bytes-repr"] = repr(value) + return forms + return {} + + +def _path_component(path: str, key: Any) -> str: + if isinstance(key, str): + key_text = key + elif isinstance(key, (bool, int, float, complex)): + key_text = str(key) + else: + # Composite/custom labels can invoke hostile ``__repr__`` methods; + # report a structural placeholder rather than serializing them. + return f"{path}[