diff --git a/README.md b/README.md index 3a1820b..957441b 100644 --- a/README.md +++ b/README.md @@ -67,6 +67,14 @@ ocr-bench bench --max-samples 50 **`ocr-bench run`** launches OCR models on your dataset via [HF Jobs](https://huggingface.co/docs/hub/jobs-overview). Each model writes its output as a PR on the same Hub dataset, keeping everything together without merge conflicts. +**`ocr-bench score`** independently computes corpus-level Character Error Rate (CER) and Word Error Rate (WER) when an output dataset has a ground-truth transcription column. It makes no model or judge calls: + +```bash +ocr-bench score --reference-column reference +``` + +Known HTML is flattened, Unicode is canonicalised, and whitespace is collapsed so line wrapping does not dominate the score; case and punctuation remain significant. Failed OCR sentinels count as empty predictions rather than being dropped. The published `metrics`, `metric_details`, and `metric_metadata` configs preserve aggregate scores and provenance. + **`ocr-bench judge`** runs pairwise comparisons using a VLM judge (default: [Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) via HF Inference Providers). For each document, the judge sees the original image and two OCR outputs (anonymised as A/B) and picks the better transcription. Results are fit to a [Bradley-Terry model](https://en.wikipedia.org/wiki/Bradley%E2%80%93Terry_model) to produce ELO ratings with bootstrap 95% confidence intervals. Adaptive stopping halts early when rankings are statistically resolved. The historical `balanced` adaptive strategy judges every model pair in each five-page batch. Large model grids can opt into targeted allocation once the run has enough balanced evidence to compute an initial board: @@ -85,6 +93,10 @@ To avoid wasting judge calls on uninformative pairs, the judge skips comparisons **`ocr-bench view`** serves a local web viewer with a leaderboard, comparison browser, and human validation. Vote on comparisons to cross-check the automated judge with human judgement. +### Kat57 ground truth + +[`experiments/kat57`](experiments/kat57) contains a bounded-memory converter for Lund University Library's [Kat57 PAGE XML release](https://zenodo.org/records/14679534). It turns the 10,695 manually transcribed catalogue cards into a Hugging Face image-and-reference dataset suitable for `ocr-bench run` and `ocr-bench score`; the completed conversion is available as [`tadad/kat57-ground-truth`](https://huggingface.co/datasets/tadad/kat57-ground-truth). The [published 500-card acceptance benchmark](https://huggingface.co/datasets/tadad/kat57-ocr-bench-500-results) reports CER/WER only and documents the limitations of applying sequence metrics to nonlinear catalogue cards. + ## Available models ocr-bench ships with 16 registered OCR models. Five form the default set: diff --git a/experiments/kat57/README.md b/experiments/kat57/README.md new file mode 100644 index 0000000..6c4b625 --- /dev/null +++ b/experiments/kat57/README.md @@ -0,0 +1,66 @@ +# Kat57 ground-truth conversion + +This experiment converts Lund University Library's 36 GB +[Kat57 Zenodo release](https://zenodo.org/records/14679534) into a data-only +Hugging Face dataset that `ocr-bench run` can consume directly. + +The source contains 10,695 PNG images paired with eScriptorium PAGE XML. The +converter verifies the published ZIP MD5, extracts human-transcribed +line text in XML order, retains the original PAGE XML, writes bounded-memory +Parquet shards, and uploads each shard before creating the next one. + +## Published artifacts + +- [`tadad/kat57-ground-truth`](https://huggingface.co/datasets/tadad/kat57-ground-truth) + is the complete conversion: 10,695 rows in 84 Parquet shards. The Hub dataset + server reports the expected row count and no pending, failed, or partial work. +- [`tadad/kat57-ocr-bench-results`](https://huggingface.co/datasets/tadad/kat57-ocr-bench-results) + is a 50-card end-to-end smoke test. Tesseract OCR outputs were scored against + the references and published as `metrics`, `metric_details`, and + `metric_metadata` configs. +- [`tadad/kat57-ocr-bench-500-results`](https://huggingface.co/datasets/tadad/kat57-ocr-bench-500-results) + is the comparative acceptance run: 16 OCR models scored on a deterministic + 500-card sample, for 8,000 reference-based evaluations. It contains CER/WER + results only; no VLM judge or ELO results are part of the Kat57 benchmark. + +Run it on a cheap CPU Job; `cpu-basic` has just enough disk because only the +source archive and one output shard coexist: + +```bash +hf jobs uv run \ + --flavor cpu-basic \ + --timeout 8h \ + --secrets HF_TOKEN \ + experiments/kat57/prepare.py \ + tadad/kat57-ground-truth +``` + +For a disposable integration test, publish a small private subset to a fresh +repository: + +```bash +hf jobs uv run \ + --flavor cpu-basic \ + --timeout 8h \ + --secrets HF_TOKEN \ + experiments/kat57/prepare.py \ + tadad/kat57-ground-truth-smoke \ + --max-samples 10 \ + --private +``` + +For a reproducible benchmark sample drawn across the full archive, combine +`--max-samples` with `--sample-seed`: + +```bash +hf jobs uv run \ + --flavor cpu-basic \ + --timeout 8h \ + --secrets HF_TOKEN \ + experiments/kat57/prepare.py \ + tadad/kat57-ground-truth-500 \ + --max-samples 500 \ + --sample-seed 57 +``` + +The converter refuses to overwrite a repository that already contains data. diff --git a/experiments/kat57/prepare.py b/experiments/kat57/prepare.py new file mode 100644 index 0000000..57bdbbe --- /dev/null +++ b/experiments/kat57/prepare.py @@ -0,0 +1,205 @@ +# /// script +# requires-python = ">=3.11" +# dependencies = ["datasets>=4.0.0", "huggingface-hub>=0.30", "pillow"] +# /// +"""Convert Kat57's paired PNG/PAGE XML archive to a Hugging Face dataset.""" + +from __future__ import annotations + +import argparse +import hashlib +import os +import urllib.request +import zipfile +from dataclasses import dataclass +from pathlib import Path +from xml.etree import ElementTree + +from datasets import Dataset, Features, Image, Sequence, Value +from huggingface_hub import HfApi, get_token + +ARCHIVE_URL = "https://zenodo.org/api/records/14679534/files/kat57-gt-dataset.zip/content" +ARCHIVE_MD5 = "0895d5785cd0ddf9f9c6488484ac447e" + +FEATURES = Features( + { + "id": Value("string"), + "drawer_id": Value("string"), + "card_id": Value("string"), + "image": Image(), + "reference": Value("string"), + "lines": Sequence(Value("string")), + "line_count": Value("int32"), + "page_xml": Value("string"), + "source_image": Value("string"), + } +) + + +@dataclass(frozen=True) +class CardPair: + card_id: str + image_member: str + xml_member: str + + +def parse_page_xml(xml_bytes: bytes) -> tuple[list[str], str]: + """Extract each PAGE TextLine transcription in document order.""" + root = ElementTree.fromstring(xml_bytes) + lines = [] + for element in root.iter(): + if element.tag.rsplit("}", 1)[-1] != "TextLine": + continue + text = next( + ( + child.text or "" + for child in element.iter() + if child.tag.rsplit("}", 1)[-1] == "Unicode" + ), + "", + ) + if text: + lines.append(text) + return lines, "\n".join(lines) + + +def discover_pairs(archive: zipfile.ZipFile) -> list[CardPair]: + """Match PNG and PAGE XML members by filename stem.""" + members: dict[str, dict[str, str]] = {".png": {}, ".xml": {}} + for name in archive.namelist(): + suffix = Path(name).suffix.lower() + if suffix in members and not name.startswith("__MACOSX/"): + members[suffix][Path(name).stem] = name + + images, xml_files = members[".png"], members[".xml"] + if not images or images.keys() != xml_files.keys(): + raise ValueError( + f"Kat57 archive has {len(images)} PNGs and {len(xml_files)} PAGE XML files" + ) + return [CardPair(stem, images[stem], xml_files[stem]) for stem in sorted(images)] + + +def deterministic_sample(pairs: list[CardPair], size: int, seed: int) -> list[CardPair]: + """Choose a stable seeded sample and return it in source order.""" + if not 1 <= size <= len(pairs): + raise ValueError(f"Sample size must be between 1 and {len(pairs)}") + chosen = sorted( + pairs, + key=lambda pair: hashlib.sha256(f"{seed}\0{pair.card_id}".encode()).digest(), + )[:size] + return sorted(chosen, key=lambda pair: pair.card_id) + + +def card_row(archive: zipfile.ZipFile, pair: CardPair) -> dict[str, object]: + xml_bytes = archive.read(pair.xml_member) + lines, reference = parse_page_xml(xml_bytes) + parts = pair.card_id.split("-") + return { + "id": pair.card_id, + "drawer_id": parts[-2], + "card_id": parts[-1], + "image": { + "bytes": archive.read(pair.image_member), + "path": Path(pair.image_member).name, + }, + "reference": reference, + "lines": lines, + "line_count": len(lines), + "page_xml": xml_bytes.decode(), + "source_image": Path(pair.image_member).name, + } + + +def md5sum(path: Path) -> str: + digest = hashlib.md5() # noqa: S324 - compare with Zenodo's published checksum + with path.open("rb") as file: + for chunk in iter(lambda: file.read(8 * 1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def download_archive(destination: Path) -> Path: + destination.parent.mkdir(parents=True, exist_ok=True) + print(f"Downloading {ARCHIVE_URL}") + urllib.request.urlretrieve(ARCHIVE_URL, destination) # noqa: S310 - fixed HTTPS URL + return destination + + +def publish( + archive_path: Path, + repo_id: str, + *, + shard_size: int, + max_samples: int | None, + sample_seed: int, + private: bool, + work_dir: Path, +) -> None: + """Upload small Parquet shards so the 36 GB source fits on a CPU Job.""" + if md5sum(archive_path) != ARCHIVE_MD5: + raise ValueError("Archive checksum does not match the Kat57 Zenodo release") + + token = get_token() or os.environ.get("HF_TOKEN") + if not token: + raise RuntimeError("Authenticate with `hf auth login` or HF_TOKEN") + api = HfApi(token=token) + api.create_repo(repo_id, repo_type="dataset", private=private, exist_ok=True) + if any(path.startswith("data/") for path in api.list_repo_files(repo_id, repo_type="dataset")): + raise RuntimeError(f"{repo_id} already contains data; choose an empty repository") + + work_dir.mkdir(parents=True, exist_ok=True) + with zipfile.ZipFile(archive_path) as archive: + pairs = discover_pairs(archive) + if max_samples is not None: + pairs = deterministic_sample(pairs, max_samples, sample_seed) + shard_count = (len(pairs) + shard_size - 1) // shard_size + for shard_index, start in enumerate(range(0, len(pairs), shard_size)): + rows = [card_row(archive, pair) for pair in pairs[start : start + shard_size]] + local_path = work_dir / f"train-{shard_index:05d}-of-{shard_count:05d}.parquet" + Dataset.from_list(rows, features=FEATURES).to_parquet(local_path) + api.upload_file( + path_or_fileobj=local_path, + path_in_repo=f"data/{local_path.name}", + repo_id=repo_id, + repo_type="dataset", + ) + local_path.unlink() + print(f"Uploaded shard {shard_index + 1}/{shard_count}") + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("repo_id") + parser.add_argument("--archive", type=Path) + parser.add_argument("--work-dir", type=Path, default=Path("kat57-work")) + parser.add_argument("--shard-size", type=int, default=128) + parser.add_argument("--max-samples", type=int) + parser.add_argument("--sample-seed", type=int, default=57) + parser.add_argument("--private", action="store_true") + args = parser.parse_args() + + if args.shard_size < 1 or (args.max_samples is not None and args.max_samples < 1): + parser.error("shard size and max samples must be positive") + archive = args.archive or args.work_dir / "kat57-gt-dataset.zip" + downloaded = args.archive is None + if downloaded: + download_archive(archive) + elif not archive.is_file(): + parser.error(f"archive not found: {archive}") + try: + publish( + archive, + args.repo_id, + shard_size=args.shard_size, + max_samples=args.max_samples, + sample_seed=args.sample_seed, + private=args.private, + work_dir=args.work_dir, + ) + finally: + if downloaded and archive.exists(): + archive.unlink() + + +if __name__ == "__main__": + main() diff --git a/pyproject.toml b/pyproject.toml index 79d138c..3b54b42 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -14,6 +14,7 @@ dependencies = [ "numpy", "openai", "pillow", + "rapidfuzz>=3.0", "rich", "scipy", "stamina", diff --git a/src/ocr_bench/cli.py b/src/ocr_bench/cli.py index 41de068..23c6868 100644 --- a/src/ocr_bench/cli.py +++ b/src/ocr_bench/cli.py @@ -10,6 +10,7 @@ from typing import TYPE_CHECKING import structlog +from datasets import Dataset from openai import OpenAIError from rich.console import Console from rich.table import Table @@ -56,11 +57,13 @@ sample_indices, validate_prompt_template, ) +from ocr_bench.metrics import MetricResult, score_dataset from ocr_bench.publish import ( EvalMetadata, load_existing_comparisons, load_existing_metadata, publish_checkpoint, + publish_metric_results, publish_results, ) @@ -298,6 +301,38 @@ def build_parser() -> argparse.ArgumentParser: help="Number of concurrent judge API calls (default: 1)", ) + # --- score subcommand --- + score = sub.add_parser( + "score", help="Compute CER/WER against ground-truth transcriptions" + ) + score.add_argument("dataset", help="HF dataset repo id with OCR outputs") + score.add_argument( + "--reference-column", + required=True, + help="Column containing ground-truth transcriptions", + ) + score.add_argument("--split", default="train", help="Dataset split (default: train)") + score.add_argument("--columns", nargs="+", default=None, help="Explicit OCR column names") + score.add_argument( + "--configs", nargs="+", default=None, help="Config-per-model: list of config names" + ) + score.add_argument( + "--from-prs", action="store_true", help="Force PR-based config discovery" + ) + score.add_argument( + "--merge", + action="store_true", + help="Merge PRs to main after discovery (default: load via revision)", + ) + score.add_argument( + "--save-results", + default=None, + help="HF repo for metric results (default: {dataset}-results)", + ) + score.add_argument( + "--no-publish", action="store_true", help="Print scores without publishing" + ) + # --- run subcommand --- run = sub.add_parser("run", help="Launch OCR models on a dataset via HF Jobs") run.add_argument( @@ -803,13 +838,131 @@ def _existing_preprocessing_provenance(meta_rows: list[dict]) -> tuple[str, int, ) +def _load_evaluation_dataset( + args: argparse.Namespace, +) -> tuple[Dataset, dict[str, str], bool]: + """Load OCR outputs for judge/score using one shared discovery policy. + + Returns ``(dataset, ocr_columns, from_prs)`` so both commands support the + same flat, config-based, and pull-request-backed dataset layouts. + """ + merge = args.merge + from_prs = False + + if args.configs: + ds, ocr_columns = load_config_dataset( + args.dataset, args.configs, split=args.split + ) + elif args.columns: + ds, ocr_columns = load_flat_dataset( + args.dataset, split=args.split, columns=args.columns + ) + elif args.from_prs: + config_names, pr_revisions = discover_pr_configs(args.dataset, merge=merge) + if not config_names: + raise DatasetError("No configs found in open PRs") + from_prs = True + console.print(f"Discovered {len(config_names)} configs from PRs: {config_names}") + ds, ocr_columns = load_config_dataset( + args.dataset, + config_names, + split=args.split, + pr_revisions=pr_revisions if not merge else None, + ) + else: + pr_configs, pr_revisions = discover_pr_configs(args.dataset, merge=merge) + main_configs = discover_configs(args.dataset) + config_names = list(pr_configs) + for main_config in main_configs: + if main_config not in pr_configs: + config_names.append(main_config) + + if config_names: + if pr_configs: + from_prs = True + console.print( + f"Auto-detected {len(pr_configs)} configs from PRs: {pr_configs}" + ) + main_only = [config for config in main_configs if config not in pr_configs] + if main_only: + console.print( + f"Auto-detected {len(main_only)} configs on main: {main_only}" + ) + ds, ocr_columns = load_config_dataset( + args.dataset, + config_names, + split=args.split, + pr_revisions=pr_revisions if pr_configs else None, + ) + else: + ds, ocr_columns = load_flat_dataset(args.dataset, split=args.split) + + return ds, ocr_columns, from_prs + + +def print_metric_leaderboard(result: MetricResult) -> None: + """Print a corpus-level CER/WER table, best scores first.""" + table = Table(title="OCR Ground-Truth Metrics") + table.add_column("Rank", style="bold") + table.add_column("Model") + table.add_column("CER", justify="right") + table.add_column("WER", justify="right") + table.add_column("Samples", justify="right") + table.add_column("Failed", justify="right") + table.add_column("Skipped", justify="right") + + for rank, summary in enumerate(result.summaries, 1): + table.add_row( + str(rank), + summary.model, + f"{summary.cer:.4f}", + f"{summary.wer:.4f}", + str(summary.evaluated_samples), + str(summary.failed_outputs), + str(summary.skipped_samples), + ) + console.print(table) + console.print(f"[dim]Reference: {result.reference_column}; lower is better.[/dim]") + + +def cmd_score(args: argparse.Namespace) -> None: + """Load OCR outputs, compute ground-truth metrics, print, and publish.""" + ds, ocr_columns, from_prs = _load_evaluation_dataset(args) + try: + result = score_dataset( + ds, + ocr_columns, + args.reference_column, + ) + except ValueError as exc: + raise DatasetError(str(exc)) from exc + + if not any(summary.evaluated_samples for summary in result.summaries): + raise DatasetError( + f"Reference column '{args.reference_column}' has no non-empty rows to score" + ) + + console.print(f"Loaded {len(ds)} samples with {len(result.summaries)} models:") + for summary in result.summaries: + console.print(f" {summary.column} → {summary.model}") + print_metric_leaderboard(result) + results_repo = _resolve_results_repo(args.dataset, args.save_results, args.no_publish) + if results_repo: + publish_metric_results( + results_repo, + result, + source_dataset=args.dataset, + source_split=args.split, + from_prs=from_prs, + ) + console.print(f"\nMetrics published to [bold]{results_repo}[/bold]") + + def cmd_judge(args: argparse.Namespace) -> None: """Orchestrate: load → compare → judge → elo → print → publish.""" # --- Resolve flags --- adaptive = not args.no_adaptive - merge = args.merge results_repo = _resolve_results_repo(args.dataset, args.save_results, args.no_publish) - from_prs = False # track for metadata max_comparisons = args.max_comparisons # global budget; None = uncapped adaptive_strategy = args.adaptive_strategy if not adaptive and adaptive_strategy != "balanced": @@ -871,55 +1024,8 @@ def cmd_judge(args: argparse.Namespace) -> None: if results_repo: console.print(f"Results will be published to [bold]{results_repo}[/bold]") - # --- Load dataset (cascading auto-detection) --- - if args.configs: - # Explicit configs — use them directly - config_names = args.configs - ds, ocr_columns = load_config_dataset(args.dataset, config_names, split=args.split) - elif args.columns: - # Explicit columns — flat loading - ds, ocr_columns = load_flat_dataset(args.dataset, split=args.split, columns=args.columns) - elif args.from_prs: - # Forced PR discovery - config_names, pr_revisions = discover_pr_configs(args.dataset, merge=merge) - if not config_names: - raise DatasetError("No configs found in open PRs") - from_prs = True - console.print(f"Discovered {len(config_names)} configs from PRs: {config_names}") - ds, ocr_columns = load_config_dataset( - args.dataset, - config_names, - split=args.split, - pr_revisions=pr_revisions if not merge else None, - ) - else: - # Auto-detect: PRs + main branch configs combined, fall back to flat - pr_configs, pr_revisions = discover_pr_configs(args.dataset, merge=merge) - main_configs = discover_configs(args.dataset) - - # Combine: PR configs + main configs not already in PRs - config_names = list(pr_configs) - for mc in main_configs: - if mc not in pr_configs: - config_names.append(mc) - - if config_names: - if pr_configs: - from_prs = True - console.print(f"Auto-detected {len(pr_configs)} configs from PRs: {pr_configs}") - if main_configs: - main_only = [c for c in main_configs if c not in pr_configs] - if main_only: - console.print(f"Auto-detected {len(main_only)} configs on main: {main_only}") - ds, ocr_columns = load_config_dataset( - args.dataset, - config_names, - split=args.split, - pr_revisions=pr_revisions if pr_configs else None, - ) - else: - # No configs anywhere — fall back to flat loading - ds, ocr_columns = load_flat_dataset(args.dataset, split=args.split) + # --- Load dataset (shared cascading auto-detection) --- + ds, ocr_columns, from_prs = _load_evaluation_dataset(args) console.print(f"Loaded {len(ds)} samples with {len(ocr_columns)} models:") for col, model in ocr_columns.items(): @@ -1992,6 +2098,8 @@ def main() -> None: try: if args.command == "judge": cmd_judge(args) + elif args.command == "score": + cmd_score(args) elif args.command == "run": cmd_run(args) elif args.command == "view": diff --git a/src/ocr_bench/metrics.py b/src/ocr_bench/metrics.py new file mode 100644 index 0000000..8aeed50 --- /dev/null +++ b/src/ocr_bench/metrics.py @@ -0,0 +1,202 @@ +"""Ground-truth OCR metrics. + +Character and word error rates complement the pairwise VLM judge when a +collection includes reference transcriptions. The implementation records the +edit-distance numerators and denominators so published scores remain auditable. +""" + +from __future__ import annotations + +import html +import re +import unicodedata +from collections.abc import Sequence +from dataclasses import dataclass +from typing import TypeVar + +from datasets import Dataset +from rapidfuzz.distance import Levenshtein + +from ocr_bench.judge import is_sentinel, normalize_for_judge + +_WHITESPACE_RE = re.compile(r"\s+") +_T = TypeVar("_T") + + +@dataclass(frozen=True) +class MetricSummary: + """Corpus-level error totals for one OCR model.""" + + model: str + column: str + char_errors: int + reference_chars: int + word_errors: int + reference_words: int + evaluated_samples: int + skipped_samples: int + failed_outputs: int + + @property + def cer(self) -> float: + return self.char_errors / self.reference_chars if self.reference_chars else 0.0 + + @property + def wer(self) -> float: + return self.word_errors / self.reference_words if self.reference_words else 0.0 + + def as_row(self) -> dict[str, object]: + """Return the stable aggregate schema used by CLI and Hub publishing.""" + return { + "model": self.model, + "column": self.column, + "cer": self.cer, + "wer": self.wer, + "char_errors": self.char_errors, + "reference_chars": self.reference_chars, + "word_errors": self.word_errors, + "reference_words": self.reference_words, + "evaluated_samples": self.evaluated_samples, + "skipped_samples": self.skipped_samples, + "failed_outputs": self.failed_outputs, + } + + +@dataclass(frozen=True) +class MetricResult: + """Aggregate and per-sample scores for a collection.""" + + summaries: list[MetricSummary] + details: list[dict[str, object]] + reference_column: str + + +def normalize_metric_text(value: object) -> str: + """Normalize Unicode, HTML, and whitespace while preserving case/punctuation.""" + if value is None: + text = "" + elif isinstance(value, str): + text = value + elif isinstance(value, (bytes, bytearray)): + text = value.decode("utf-8", "replace") + else: + text = str(value) + + text = unicodedata.normalize("NFC", text) + return _WHITESPACE_RE.sub(" ", html.unescape(normalize_for_judge(text))).strip() + + +def levenshtein_distance(reference: Sequence[_T], prediction: Sequence[_T]) -> int: + """Return unit-cost Levenshtein distance for text or token sequences.""" + return Levenshtein.distance(reference, prediction) + + +def score_dataset( + dataset: Dataset, + ocr_columns: dict[str, str], + reference_column: str, +) -> MetricResult: + """Score every OCR column against a reference transcription column. + + Empty reference rows are skipped because CER/WER have no meaningful + denominator. Error sentinels are evaluated as empty predictions, ensuring + failed inference is penalized rather than making a model look better by + silently dropping its hardest pages. + """ + if reference_column not in dataset.column_names: + raise ValueError( + f"Reference column '{reference_column}' not found. " + f"Available columns: {dataset.column_names}" + ) + # Flat-dataset discovery intentionally treats names such as ``text`` and + # ``ocr_*`` as likely model outputs. Ground-truth columns commonly use those + # names too, so exclude the declared reference even if discovery selected it. + scored_columns = { + column: model for column, model in ocr_columns.items() if column != reference_column + } + if not scored_columns: + raise ValueError( + f"No OCR output columns remain after excluding reference column " + f"'{reference_column}'" + ) + missing = [column for column in scored_columns if column not in dataset.column_names] + if missing: + raise ValueError(f"OCR columns not found: {', '.join(missing)}") + + references = dataset[reference_column] + identifiers = _identifier_values(dataset) + summaries: list[MetricSummary] = [] + details: list[dict[str, object]] = [] + + for column, model in scored_columns.items(): + predictions = dataset[column] + char_errors = reference_chars = 0 + word_errors = reference_words = 0 + evaluated = skipped = failed = 0 + + for sample_idx in range(len(dataset)): + reference_value = references[sample_idx] + prediction_value = predictions[sample_idx] + reference = normalize_metric_text(reference_value) + if not reference.strip(): + skipped += 1 + continue + + prediction_raw = "" if prediction_value is None else str(prediction_value) + prediction_failed = is_sentinel(prediction_raw) + prediction = "" if prediction_failed else normalize_metric_text(prediction_value) + ref_words = reference.split() + pred_words = prediction.split() + sample_char_errors = levenshtein_distance(reference, prediction) + sample_word_errors = levenshtein_distance(ref_words, pred_words) + + evaluated += 1 + failed += int(prediction_failed) + char_errors += sample_char_errors + reference_chars += len(reference) + word_errors += sample_word_errors + reference_words += len(ref_words) + details.append( + { + "sample_idx": sample_idx, + "id": identifiers[sample_idx], + "model": model, + "column": column, + "cer": sample_char_errors / len(reference), + "wer": sample_word_errors / len(ref_words), + "char_errors": sample_char_errors, + "reference_chars": len(reference), + "word_errors": sample_word_errors, + "reference_words": len(ref_words), + "failed_output": prediction_failed, + } + ) + + summaries.append( + MetricSummary( + model=model, + column=column, + char_errors=char_errors, + reference_chars=reference_chars, + word_errors=word_errors, + reference_words=reference_words, + evaluated_samples=evaluated, + skipped_samples=skipped, + failed_outputs=failed, + ) + ) + + summaries.sort(key=lambda summary: (summary.cer, summary.wer, summary.model)) + return MetricResult( + summaries=summaries, + details=details, + reference_column=reference_column, + ) + + +def _identifier_values(dataset: Dataset) -> list[str]: + """Return stable display identifiers without decoding image rows.""" + for column in ("id", "source_row", "b_number", "page_index"): + if column in dataset.column_names: + return [str(value) if value is not None else "" for value in dataset[column]] + return [str(index) for index in range(len(dataset))] diff --git a/src/ocr_bench/publish.py b/src/ocr_bench/publish.py index 52543cd..8dcd803 100644 --- a/src/ocr_bench/publish.py +++ b/src/ocr_bench/publish.py @@ -20,6 +20,7 @@ ) from ocr_bench.elo import ComparisonResult, Leaderboard, compute_elo from ocr_bench.judge import MAX_IMAGE_DIM, MAX_OCR_TEXT_LENGTH +from ocr_bench.metrics import MetricResult from ocr_bench.run import MODEL_REGISTRY logger = structlog.get_logger() @@ -166,6 +167,61 @@ def load_existing_metadata(repo_id: str) -> list[dict]: ) from exc +def load_existing_metric_metadata(repo_id: str) -> list[dict]: + """Load the append-only metric run log, failing closed on uncertain reads.""" + try: + ds = load_dataset(repo_id, name="metric_metadata", split="train") + return [dict(row) for row in ds] + except Exception as exc: + if _config_is_absent(repo_id, "metric_metadata"): + logger.info("no_existing_metric_metadata", repo=repo_id, reason=str(exc)) + return [] + raise OSError( + f"Existing metric metadata in {repo_id} could not be loaded; refusing " + "to overwrite Hub history" + ) from exc + + +def publish_metric_results( + repo_id: str, + result: MetricResult, + *, + source_dataset: str, + source_split: str, + from_prs: bool, +) -> None: + """Publish aggregate and per-sample CER/WER with run provenance.""" + # Verify the append-only history is readable before replacing any configs. + # Otherwise a transient metadata read failure could leave new scores without + # the provenance row that explains how they were produced. + metadata_rows = load_existing_metric_metadata(repo_id) + + aggregate_rows = [summary.as_row() for summary in result.summaries] + Dataset.from_list(aggregate_rows).push_to_hub(repo_id, config_name="metrics") + logger.info("published_metrics", repo=repo_id, n=len(aggregate_rows)) + + if result.details: + Dataset.from_list(result.details).push_to_hub( + repo_id, config_name="metric_details" + ) + logger.info("published_metric_details", repo=repo_id, n=len(result.details)) + + metadata_rows.append( + { + "source_dataset": source_dataset, + "source_split": source_split, + "reference_column": result.reference_column, + "from_prs": from_prs, + "normalization": "NFC, flattened HTML, collapsed whitespace", + "timestamp": datetime.datetime.now(datetime.UTC).isoformat(), + } + ) + Dataset.from_list(_align_metadata_rows(metadata_rows)).push_to_hub( + repo_id, config_name="metric_metadata" + ) + logger.info("published_metric_metadata", repo=repo_id, n=len(metadata_rows)) + + def _get_model_sizes() -> dict[str, str]: """Build model_id → size lookup from the model registry.""" return {cfg.model_id: cfg.size for cfg in MODEL_REGISTRY.values()} diff --git a/tests/test_cli.py b/tests/test_cli.py index ad4e5b6..b3a01a5 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -115,6 +115,20 @@ def test_merge_flag(self): args = parser.parse_args(["judge", "user/dataset", "--merge"]) assert args.merge is True + def test_score_subcommand_defaults(self): + args = build_parser().parse_args( + ["score", "user/dataset", "--reference-column", "reference"] + ) + assert args.command == "score" + assert args.dataset == "user/dataset" + assert args.reference_column == "reference" + assert args.split == "train" + assert args.no_publish is False + + def test_score_requires_reference_column(self): + with pytest.raises(SystemExit): + build_parser().parse_args(["score", "user/dataset"]) + def test_no_adaptive_flag(self): parser = build_parser() args = parser.parse_args(["judge", "user/dataset", "--no-adaptive"]) @@ -990,6 +1004,88 @@ def test_failed_job_flagged_and_no_eval_hint(self, monkeypatch, capsys): assert "Evaluate:" not in out # don't suggest judging a partial run +class TestCmdScore: + def test_scores_and_prints_without_publishing(self, monkeypatch, capsys): + from datasets import Dataset + + dataset = Dataset.from_dict( + {"reference": ["hello world"], "ocr": ["hello world"]} + ) + monkeypatch.setattr( + cli, + "load_flat_dataset", + lambda *args, **kwargs: (dataset, {"ocr": "model-a"}), + ) + publish = MagicMock() + monkeypatch.setattr(cli, "publish_metric_results", publish) + args = build_parser().parse_args( + [ + "score", + "user/dataset", + "--reference-column", + "reference", + "--columns", + "ocr", + "--no-publish", + ] + ) + + cli.cmd_score(args) + + assert "OCR Ground-Truth Metrics" in capsys.readouterr().out + publish.assert_not_called() + + def test_publishes_to_shared_results_repo(self, monkeypatch): + from datasets import Dataset + + dataset = Dataset.from_dict({"reference": ["abc"], "ocr": ["axc"]}) + monkeypatch.setattr( + cli, + "load_flat_dataset", + lambda *args, **kwargs: (dataset, {"ocr": "model-a"}), + ) + publish = MagicMock() + monkeypatch.setattr(cli, "publish_metric_results", publish) + args = build_parser().parse_args( + [ + "score", + "user/dataset", + "--reference-column", + "reference", + "--columns", + "ocr", + ] + ) + + cli.cmd_score(args) + + assert publish.call_args.args[0] == "user/dataset-results" + assert publish.call_args.kwargs["source_dataset"] == "user/dataset" + + def test_empty_reference_fails_cleanly(self, monkeypatch): + from datasets import Dataset + + dataset = Dataset.from_dict({"reference": [""], "ocr": ["text"]}) + monkeypatch.setattr( + cli, + "load_flat_dataset", + lambda *args, **kwargs: (dataset, {"ocr": "model-a"}), + ) + args = build_parser().parse_args( + [ + "score", + "user/dataset", + "--reference-column", + "reference", + "--columns", + "ocr", + "--no-publish", + ] + ) + with pytest.raises(DatasetError, match="no non-empty rows"): + cli.cmd_score(args) + + class TestCmdJudgeEmptyGuard: def test_adaptive_all_filtered_does_not_publish_empty_board(self, monkeypatch, capsys): """Aggressive filtering can leave the adaptive loop with zero results; diff --git a/tests/test_kat57_prepare.py b/tests/test_kat57_prepare.py new file mode 100644 index 0000000..8e1d08c --- /dev/null +++ b/tests/test_kat57_prepare.py @@ -0,0 +1,92 @@ +"""Tests for the Kat57 PAGE XML conversion experiment.""" + +from __future__ import annotations + +import io +import zipfile + +import pytest +from PIL import Image as PILImage + +from experiments.kat57.prepare import ( + CardPair, + card_row, + deterministic_sample, + discover_pairs, + parse_page_xml, +) + +PAGE_XML = b""" + + + + Café + Second line + + + +""" + + +def png_bytes() -> bytes: + output = io.BytesIO() + PILImage.new("RGB", (10, 10), "white").save(output, format="PNG") + return output.getvalue() + + +def make_archive(tmp_path, *, include_image=True, include_xml=True): + path = tmp_path / "kat57.zip" + with zipfile.ZipFile(path, "w") as archive: + if include_image: + archive.writestr("kat57-gt-dataset/images/kat57-00145-00204.png", png_bytes()) + if include_xml: + archive.writestr("kat57-gt-dataset/pagexml/kat57-00145-00204.xml", PAGE_XML) + return path + + +def test_parse_page_xml_preserves_document_order(): + lines, reference = parse_page_xml(PAGE_XML) + assert lines == ["Café", "Second line"] + assert reference == "Café\nSecond line" + + +def test_discover_pairs_and_build_row(tmp_path): + with zipfile.ZipFile(make_archive(tmp_path)) as archive: + pairs = discover_pairs(archive) + assert len(pairs) == 1 + row = card_row(archive, pairs[0]) + + assert row["id"] == "kat57-00145-00204" + assert row["drawer_id"] == "00145" + assert row["card_id"] == "00204" + assert row["reference"] == "Café\nSecond line" + assert row["line_count"] == 2 + assert row["image"]["bytes"].startswith(b"\x89PNG") + + +@pytest.mark.parametrize(("include_image", "include_xml"), [(False, True), (True, False)]) +def test_discover_pairs_rejects_unmatched_members(tmp_path, include_image, include_xml): + with zipfile.ZipFile( + make_archive(tmp_path, include_image=include_image, include_xml=include_xml) + ) as archive: + with pytest.raises(ValueError, match="archive has"): + discover_pairs(archive) + + +def test_deterministic_sample_is_reproducible_and_returned_in_source_order(): + pairs = [CardPair(f"kat57-00001-{i:05d}", f"{i}.png", f"{i}.xml") for i in range(20)] + + first = deterministic_sample(pairs, 5, seed=57) + second = deterministic_sample(list(reversed(pairs)), 5, seed=57) + different = deterministic_sample(pairs, 5, seed=58) + + assert first == second + assert first == sorted(first, key=lambda pair: pair.card_id) + assert first != different + + +@pytest.mark.parametrize("sample_size", [0, 21]) +def test_deterministic_sample_rejects_invalid_size(sample_size): + pairs = [CardPair("kat57-00001-00001", "1.png", "1.xml")] + with pytest.raises(ValueError, match="Sample size"): + deterministic_sample(pairs, sample_size, seed=57) diff --git a/tests/test_metrics.py b/tests/test_metrics.py new file mode 100644 index 0000000..0c4ebde --- /dev/null +++ b/tests/test_metrics.py @@ -0,0 +1,110 @@ +"""Tests for ground-truth CER/WER scoring.""" + +from __future__ import annotations + +import pytest +from datasets import Dataset + +from ocr_bench.metrics import levenshtein_distance, normalize_metric_text, score_dataset + + +@pytest.mark.parametrize( + ("reference", "prediction", "expected"), + [ + ("", "", 0), + ("", "abc", 3), + ("abc", "", 3), + ("kitten", "sitting", 3), + (["one", "two"], ["one", "too"], 1), + ], +) +def test_levenshtein_distance(reference, prediction, expected): + assert levenshtein_distance(reference, prediction) == expected + + +def test_normalized_text_flattens_html_whitespace_and_unicode(): + decomposed = "Cafe\u0301\r\n
onetwo
" + assert normalize_metric_text(decomposed) == "Café one | two" + + +def test_normalized_text_decodes_entities_without_html_tags(): + assert normalize_metric_text("one  two") == "one two" + + +def test_score_dataset_uses_corpus_totals_and_skips_empty_references(): + dataset = Dataset.from_dict( + { + "id": ["a", "b", "c"], + "reference": ["cat", "a much longer line", ""], + "model_a": ["cut", "a much longer line", "ignored"], + } + ) + + result = score_dataset(dataset, {"model_a": "model-a"}, "reference") + summary = result.summaries[0] + + assert summary.char_errors == 1 + assert summary.reference_chars == len("cat") + len("a much longer line") + assert summary.cer == pytest.approx(1 / summary.reference_chars) + assert summary.word_errors == 1 + assert summary.reference_words == 5 + assert summary.wer == pytest.approx(0.2) + assert summary.evaluated_samples == 2 + assert summary.skipped_samples == 1 + assert len(result.details) == 2 + assert result.details[0]["id"] == "a" + + +def test_failure_sentinel_is_penalized_as_empty_prediction(): + dataset = Dataset.from_dict({"reference": ["three words here"], "ocr": ["[OCR ERROR]"]}) + summary = score_dataset(dataset, {"ocr": "broken-model"}, "reference").summaries[0] + + assert summary.failed_outputs == 1 + assert summary.cer == 1.0 + assert summary.wer == 1.0 + + +def test_scores_multiple_models_and_sorts_by_cer(): + dataset = Dataset.from_dict( + { + "reference": ["hello world"], + "bad": ["hello"], + "good": ["hello world"], + } + ) + result = score_dataset( + dataset, + {"bad": "bad-model", "good": "good-model"}, + "reference", + ) + + assert [summary.model for summary in result.summaries] == ["good-model", "bad-model"] + assert result.summaries[0].cer == 0 + assert result.summaries[0].wer == 0 + + +def test_reference_column_is_not_scored_as_a_model(): + dataset = Dataset.from_dict( + {"text": ["ground truth"], "ocr_model": ["ground trvth"]} + ) + + result = score_dataset( + dataset, + {"text": "text", "ocr_model": "actual-model"}, + "text", + ) + + assert [summary.model for summary in result.summaries] == ["actual-model"] + + +def test_reference_only_column_map_fails_cleanly(): + dataset = Dataset.from_dict({"text": ["ground truth"]}) + + with pytest.raises(ValueError, match="No OCR output columns remain"): + score_dataset(dataset, {"text": "text"}, "text") + + +def test_missing_reference_column_fails_cleanly(): + dataset = Dataset.from_dict({"ocr": ["text"]}) + with pytest.raises(ValueError, match="Reference column 'reference' not found"): + score_dataset(dataset, {"ocr": "model"}, "reference") diff --git a/tests/test_publish.py b/tests/test_publish.py index 8d38e38..35b7e5b 100644 --- a/tests/test_publish.py +++ b/tests/test_publish.py @@ -8,6 +8,7 @@ from ocr_bench.adaptive import AdjacentPairDecision from ocr_bench.elo import ComparisonResult, Leaderboard +from ocr_bench.metrics import MetricResult, MetricSummary from ocr_bench.publish import ( EvalMetadata, _align_metadata_rows, @@ -15,7 +16,9 @@ build_metadata_row, load_existing_comparisons, load_existing_metadata, + load_existing_metric_metadata, publish_checkpoint, + publish_metric_results, publish_results, ) @@ -32,6 +35,40 @@ def _make_board() -> Leaderboard: ) +def _make_metric_result() -> MetricResult: + return MetricResult( + summaries=[ + MetricSummary( + model="model-a", + column="ocr_a", + char_errors=1, + reference_chars=10, + word_errors=1, + reference_words=3, + evaluated_samples=1, + skipped_samples=0, + failed_outputs=0, + ) + ], + details=[ + { + "sample_idx": 0, + "id": "card-1", + "model": "model-a", + "column": "ocr_a", + "cer": 0.1, + "wer": 1 / 3, + "char_errors": 1, + "reference_chars": 10, + "word_errors": 1, + "reference_words": 3, + "failed_output": False, + } + ], + reference_column="reference", + ) + + class TestBuildLeaderboardRows: def test_rows_ordered_by_elo(self): board = _make_board() @@ -736,6 +773,73 @@ def test_existing_metadata_files_fail_closed_on_load_error( load_existing_metadata("user/results") +class TestPublishMetricResults: + @patch("ocr_bench.publish.load_existing_metric_metadata", return_value=[]) + @patch("ocr_bench.publish.Dataset") + def test_publishes_dedicated_configs(self, mock_dataset, _mock_existing): + aggregate_ds = MagicMock() + detail_ds = MagicMock() + metadata_ds = MagicMock() + mock_dataset.from_list.side_effect = [aggregate_ds, detail_ds, metadata_ds] + + publish_metric_results( + "user/results", + _make_metric_result(), + source_dataset="user/source", + source_split="validation", + from_prs=True, + ) + + aggregate_ds.push_to_hub.assert_called_once_with( + "user/results", config_name="metrics" + ) + detail_ds.push_to_hub.assert_called_once_with( + "user/results", config_name="metric_details" + ) + metadata_ds.push_to_hub.assert_called_once_with( + "user/results", config_name="metric_metadata" + ) + metadata_row = mock_dataset.from_list.call_args_list[-1].args[0][0] + assert metadata_row["reference_column"] == "reference" + assert metadata_row["normalization"] == "NFC, flattened HTML, collapsed whitespace" + + @patch( + "ocr_bench.publish.load_existing_metric_metadata", + side_effect=OSError("history unavailable"), + ) + @patch("ocr_bench.publish.Dataset") + def test_metadata_read_failure_happens_before_any_write( + self, mock_dataset, _mock_existing + ): + with pytest.raises(OSError, match="history unavailable"): + publish_metric_results( + "user/results", + _make_metric_result(), + source_dataset="user/source", + source_split="train", + from_prs=False, + ) + + mock_dataset.from_list.assert_not_called() + + @patch("ocr_bench.publish.HfApi") + @patch("ocr_bench.publish.load_dataset") + def test_metric_metadata_missing_is_empty(self, mock_load, mock_api_cls): + mock_load.side_effect = Exception("missing") + mock_api_cls.return_value.list_repo_files.return_value = ["README.md"] + assert load_existing_metric_metadata("user/results") == [] + + @patch("ocr_bench.publish.HfApi") + @patch("ocr_bench.publish.load_dataset") + def test_metric_metadata_existing_files_fail_closed(self, mock_load, mock_api_cls): + mock_load.side_effect = Exception("temporary failure") + mock_api_cls.return_value.list_repo_files.return_value = [ + "metric_metadata/train-00000-of-00001.parquet" + ] + with pytest.raises(OSError, match="refusing to overwrite Hub history"): + load_existing_metric_metadata("user/results") + + class TestBuildReadme: def _make_metadata(self) -> EvalMetadata: return EvalMetadata( diff --git a/uv.lock b/uv.lock index 5b04cde..40826eb 100644 --- a/uv.lock +++ b/uv.lock @@ -1082,6 +1082,7 @@ dependencies = [ { name = "numpy" }, { name = "openai" }, { name = "pillow" }, + { name = "rapidfuzz" }, { name = "rich" }, { name = "scipy" }, { name = "stamina" }, @@ -1115,6 +1116,7 @@ requires-dist = [ { name = "openai" }, { name = "pillow" }, { name = "python-multipart", marker = "extra == 'viewer'", specifier = ">=0.0.9" }, + { name = "rapidfuzz", specifier = ">=3.0" }, { name = "rich" }, { name = "scipy" }, { name = "stamina" }, @@ -1687,6 +1689,116 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/f1/12/de94a39c2ef588c7e6455cfbe7343d3b2dc9d6b6b2f40c4c6565744c873d/pyyaml-6.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b", size = 149341, upload-time = "2025-09-25T21:32:56.828Z" }, ] +[[package]] +name = "rapidfuzz" +version = "3.14.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/18/97/226c43b7b5d957bc3840ed52ea99eed261f99834c4619be7a4742cbaeafa/rapidfuzz-3.14.6.tar.gz", hash = "sha256:e13a8160d017b499ec7a2fa9d0ce1ae2e7377080815785819f966fb235d4eb60", size = 57955060, upload-time = "2026-08-30T21:45:51.097Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4c/09/144d6fcd84fadb124d282f727d197a92dc48ae279e80d4b7d23795ba164d/rapidfuzz-3.14.6-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:1c0dd0d765184366b6e213a8af3b0b3bb39dad27943bbfb193515d4ff96ac82a", size = 1975267, upload-time = "2026-08-30T21:41:54.195Z" }, + { url = "https://files.pythonhosted.org/packages/b9/8f/17985248f0f651a518b543f802fa706b7810cbe96a434a5a9dc24f99b7d2/rapidfuzz-3.14.6-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:0c61cade182f130c9903231946bd1074539121721693a918e7b70382ae802bd8", size = 1246874, upload-time = "2026-08-30T21:41:57.063Z" }, + { url = "https://files.pythonhosted.org/packages/de/8f/9cf3b552bb84911add3c86e014e8704d20ea4e274295686106dc010356ae/rapidfuzz-3.14.6-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3781cf14f9fc933d7198c2b25a8bbbd1a62b752746d5cd26de14957edc0e802f", size = 1394531, upload-time = "2026-08-30T21:41:58.745Z" }, + { url = "https://files.pythonhosted.org/packages/e3/7f/c4824d855cb1f89f8db0802b7ae22705187be55e0ab2f9873b574a0a6713/rapidfuzz-3.14.6-cp311-cp311-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:71a5bbfd00da1963f27dd1432068929694cf0e00007ae2b9c1ad2a187ec29a16", size = 1702106, upload-time = "2026-08-30T21:42:00.398Z" }, + { url = "https://files.pythonhosted.org/packages/9f/ff/556d3aefbd1f115fcda6bdf3ea578405fcaa44c233b525fda583943f3692/rapidfuzz-3.14.6-cp311-cp311-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:eabaf06ca4896c59cfd9162480f0d37a15a2304ce2efe83ae2bbcfa1cf13534e", size = 2735203, upload-time = "2026-08-30T21:42:02.115Z" }, + { url = "https://files.pythonhosted.org/packages/11/ae/a781ec62825990319483c82ef962b509e9ce22a67a9f97d63d70b2b175b9/rapidfuzz-3.14.6-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3d5d90bae3c6fb7ea34da968c9f23070e8440edb827a28b242580e0108110b14", size = 3180952, upload-time = "2026-08-30T21:42:03.918Z" }, + { url = "https://files.pythonhosted.org/packages/d9/cc/a8cdeaa64db2e914f3475551b19ea2a6187b5458b50eac707e10f1bcf9d7/rapidfuzz-3.14.6-cp311-cp311-manylinux_2_39_riscv64.whl", hash = "sha256:d6b58daadbe6974884ec39aee30cfb8bd2e126f8d03503f0069f70d5e84656a3", size = 1485205, upload-time = "2026-08-30T21:42:05.659Z" }, + { url = "https://files.pythonhosted.org/packages/09/4e/6394e8d79088124bf39a8103ac2ae166a3f62ffc67b51c4e869dfe38b6d1/rapidfuzz-3.14.6-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:ab4386ef7c2cb3e5eb46e815be49715dfcd301bb9f0a431f18da7aa0007de54f", size = 2415347, upload-time = "2026-08-30T21:42:07.847Z" }, + { url = "https://files.pythonhosted.org/packages/f0/2e/92acf13a03c45884aabe9d637c620f5b7806e56bd6f6f8d8016f95614722/rapidfuzz-3.14.6-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:33a2f7faedaa3608c4876c41b448fc786d54e6cd7c6e732f7de466319b5a73c2", size = 2819438, upload-time = "2026-08-30T21:42:09.788Z" }, + { url = "https://files.pythonhosted.org/packages/95/54/3ed4286d9ebf0b623b021970a46d7befa053dd09c85cd213bfb2ad2a0bbc/rapidfuzz-3.14.6-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:adb160a100f6122aa45c78d686e198da3f9e815d4182e0c4fe730608479f7f9c", size = 2521065, upload-time = "2026-08-30T21:42:11.923Z" }, + { url = "https://files.pythonhosted.org/packages/6a/ff/ae8ecf60ce25eab3accfe5a0c9ba6499b02c5e2ab03ee9defdf5475eb4e7/rapidfuzz-3.14.6-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:ad60297c001d15af24338440bca85dfee8710e9e3222733c906b33e89d986166", size = 3319384, upload-time = "2026-08-30T21:42:14.191Z" }, + { url = "https://files.pythonhosted.org/packages/4a/1d/d39dfc6cdc5c1d0452d4af563c678f2d5821f0df306bc3ab9502f3555690/rapidfuzz-3.14.6-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:3d5b1cfa67bbe6239a643bca1d986f8a07e0a045286c674946e1648c132baa46", size = 4297470, upload-time = "2026-08-30T21:42:16.667Z" }, + { url = "https://files.pythonhosted.org/packages/1b/f6/0a64983c5cf5b2ce8cf2ce4fc54ecd6b5ee6cd6a3af8b870657f28e31a07/rapidfuzz-3.14.6-cp311-cp311-win32.whl", hash = "sha256:46ddb42af4cad3ac9d5e0c97ee1e687500c529a1ad5cbf9c949ce35f6edd4537", size = 1902086, upload-time = "2026-08-30T21:42:18.576Z" }, + { url = "https://files.pythonhosted.org/packages/41/72/638db21d63041ba17c4ed482a8cd1fe6dc4d90bc84b2a28aaccc2611ff84/rapidfuzz-3.14.6-cp311-cp311-win_amd64.whl", hash = "sha256:737a57cbca3e5c16decac86e205727bcd4b99c52f77c48bb44123078c5cd9a7a", size = 1738042, upload-time = "2026-08-30T21:42:20.427Z" }, + { url = "https://files.pythonhosted.org/packages/10/f7/d0fb82451c1f0c701a742939120b32a092ac64bbacf8bf8fa21d61fc89e7/rapidfuzz-3.14.6-cp311-cp311-win_arm64.whl", hash = "sha256:19c1cda8198cc57ffd4ff69a1c02cbe4297e9ca7b506bca03ec584da0a9fe1ff", size = 1190829, upload-time = "2026-08-30T21:42:22.322Z" }, + { url = "https://files.pythonhosted.org/packages/03/d2/5a7646b185a61400220e4783d23461c1e864a9ee82ba443b18c218e2364b/rapidfuzz-3.14.6-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b46cecf27025e7a934332ade033e6a394da8a493f19fa1d835e3b2968a4ff7da", size = 1965178, upload-time = "2026-08-30T21:42:24.164Z" }, + { url = "https://files.pythonhosted.org/packages/8b/72/10fc4e414eeed7963e2f1c315c731cb68196f0478cb244c78a21f5ce8662/rapidfuzz-3.14.6-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1901414b135afb1a7f4b1ef940b95523b49cc5642aecf02af740f37567e98137", size = 1248230, upload-time = "2026-08-30T21:42:26.088Z" }, + { url = "https://files.pythonhosted.org/packages/39/e9/0794043c1a0af09cacdbb6a9e8b9b2079cdf73337e7c29b4a9f117415bb9/rapidfuzz-3.14.6-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:96a548979cd939b2c69358a0f5088a408524fbf7454f04bf90939fa971e64310", size = 1380396, upload-time = "2026-08-30T21:42:27.97Z" }, + { url = "https://files.pythonhosted.org/packages/2f/73/9218cf4424ab86260ee88ebdb612c5ed4d9bfd6b6d1e2f3c3bf4599d13bf/rapidfuzz-3.14.6-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b22ef7e5e2341efc6216b666491022027b984e5aef93446064742f43f3c1d926", size = 1674037, upload-time = "2026-08-30T21:42:29.754Z" }, + { url = "https://files.pythonhosted.org/packages/a1/f5/bad528b6dfc608a48838508f270c79332ab05592703c9a46504ba95e9eab/rapidfuzz-3.14.6-cp312-cp312-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f0d2d95c787d812b9106cfbcb94ad37a49f59df9287e00a75eb61afc246e8759", size = 2722897, upload-time = "2026-08-30T21:42:31.737Z" }, + { url = "https://files.pythonhosted.org/packages/13/da/49ab137f788a0e03e872d4c6b3d5c9c6c6bed4e4ccea381f69c4d186341b/rapidfuzz-3.14.6-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0debb5f43662ea84d2f0228a0c7407ff647f9c3d13f3b692efff0cde46eebce0", size = 3168023, upload-time = "2026-08-30T21:42:33.663Z" }, + { url = "https://files.pythonhosted.org/packages/59/33/81ca664a15194b8b4a7e863b534e36c057724f9709c7781e9400d0edf024/rapidfuzz-3.14.6-cp312-cp312-manylinux_2_39_riscv64.whl", hash = "sha256:1d253e1fe44648242a0029b42ba23adf238ed2a7eb3d8ed0a03731a23f074ae0", size = 1474666, upload-time = "2026-08-30T21:42:35.5Z" }, + { url = "https://files.pythonhosted.org/packages/87/eb/b16f9f8cc255c8dc7c0d7712aa7e7c12a6fd85c8b2b56665f2a24222a941/rapidfuzz-3.14.6-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:e06c6050c9bf6cd72305e3e6a293918b2b92cf2a067007585a53898624902e3c", size = 2402289, upload-time = "2026-08-30T21:42:37.309Z" }, + { url = "https://files.pythonhosted.org/packages/4a/73/eaa1ca89f6ab12c0fe7f943226ce4ad1d2c67eb281dfd706279771fcff5a/rapidfuzz-3.14.6-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:d85a6e9180e53cde95c95dfeb05a2ac94ead4d9d803a8fd186d2719a678b8483", size = 2788332, upload-time = "2026-08-30T21:42:39.412Z" }, + { url = "https://files.pythonhosted.org/packages/5d/ad/db927fbe23f621dd292a6332a19822703084617c0281a88156a8c138d4e0/rapidfuzz-3.14.6-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:35db2670f69fa3a4eb4741055581477ff92f2cf39e7e06f43ebcb97c2192fe7c", size = 2510540, upload-time = "2026-08-30T21:42:41.629Z" }, + { url = "https://files.pythonhosted.org/packages/2d/b2/8e9012968fab837babe1292edcbe1c972605f5b3af19c7fcac2ded731d39/rapidfuzz-3.14.6-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:f9d93e5424d1e4c103b57906b8beba270e680afda3ffdff7ea3bc6173b37083c", size = 3299876, upload-time = "2026-08-30T21:42:43.803Z" }, + { url = "https://files.pythonhosted.org/packages/19/99/799ce99328ea97fe5d7510048ffea148b8ad4a838366f908691be52342a5/rapidfuzz-3.14.6-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:f9b0a501f37fb852c54469375baa25874246b3bbc8b6e21fb4cd186a32335868", size = 4277032, upload-time = "2026-08-30T21:42:46.08Z" }, + { url = "https://files.pythonhosted.org/packages/07/8a/995b4746c5bc1f561e64de1fa546927183fec7a369fe988716ef394a6d0a/rapidfuzz-3.14.6-cp312-cp312-win32.whl", hash = "sha256:9e974251a9833791bc557b46f975676a56c2d58946f795cd2964b095496dfdcc", size = 1887051, upload-time = "2026-08-30T21:42:48.265Z" }, + { url = "https://files.pythonhosted.org/packages/84/c4/12f01df5778227c8655fcd9b429fc001d43270f5d8d154edc9066bab1de3/rapidfuzz-3.14.6-cp312-cp312-win_amd64.whl", hash = "sha256:cfca36e4612208875e08611a779164b6cb8900ab8bbd3d82d4cfdfae9efbfac9", size = 1731992, upload-time = "2026-08-30T21:42:50.211Z" }, + { url = "https://files.pythonhosted.org/packages/19/8d/92217f0bc81ec458b4134ad53714b1be0cd3be21494227d73510b06467d6/rapidfuzz-3.14.6-cp312-cp312-win_arm64.whl", hash = "sha256:96bbd5a1c67d135334d02fae74f1d933fdda204ea03d544a59dab6b1cbfbf565", size = 1186693, upload-time = "2026-08-30T21:42:52.63Z" }, + { url = "https://files.pythonhosted.org/packages/a0/ad/4901a37256bc5027f3873ebd538b851349d7627d8aa2e91743c79b500f48/rapidfuzz-3.14.6-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:55dc9a55924b4ecfcf4a60a701bcfae7d9daf0129c41dc16139270d75be0996c", size = 1961301, upload-time = "2026-08-30T21:42:54.46Z" }, + { url = "https://files.pythonhosted.org/packages/b9/d3/5a56e26db79c00191bc7c5387a04dfa5b6326c2c81c468a976ee2aa8fa15/rapidfuzz-3.14.6-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:bba0e9fad4dbea80227cde9cef3aaa984a934a84aec5f7505532e19838b14769", size = 1244370, upload-time = "2026-08-30T21:42:56.425Z" }, + { url = "https://files.pythonhosted.org/packages/2b/12/0958686418e596961642c41e9162906363649e70f6a12cfcff212f77ccb3/rapidfuzz-3.14.6-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0b34b7ee4f4f760690d6477163aabbec05705b5dd764cb6c3a6ba95aa1fffc42", size = 1377336, upload-time = "2026-08-30T21:42:58.687Z" }, + { url = "https://files.pythonhosted.org/packages/60/09/a0a70c35996fa5225c8cddca38e2e594c82518aeefa08edb5d875ce0d82b/rapidfuzz-3.14.6-cp313-cp313-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:abe92a70134c8b40790bb5c78b2a0a790686c26e83b6e99a456127ca141fe06a", size = 1670277, upload-time = "2026-08-30T21:43:00.798Z" }, + { url = "https://files.pythonhosted.org/packages/9f/d7/b9deea614b32e933e37d77eecf539ffe2b41c0a922a6fd759993865e7ee5/rapidfuzz-3.14.6-cp313-cp313-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:659b41570fcc6e02631ac361c47cc8db9ad26d740e4be2177df1b63005a49174", size = 2722260, upload-time = "2026-08-30T21:43:02.655Z" }, + { url = "https://files.pythonhosted.org/packages/70/42/4bf9dc905df33bb4515895ff87f777d8df25a3617c0bf8f5d4716813d9ea/rapidfuzz-3.14.6-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6bb896f89a387219c671ebc33c4a636b222010cc3c5c83884a7fc8707bf0bbf9", size = 3165730, upload-time = "2026-08-30T21:43:04.632Z" }, + { url = "https://files.pythonhosted.org/packages/25/76/454acc3abfa6b958511d6e761f5a95e6c3128936a1eed4f23643c3267d8b/rapidfuzz-3.14.6-cp313-cp313-manylinux_2_39_riscv64.whl", hash = "sha256:11d76bb2b2cd038df708ae18f521fb3a50af477cc5a0dffce812da43a2f1beb3", size = 1469515, upload-time = "2026-08-30T21:43:06.612Z" }, + { url = "https://files.pythonhosted.org/packages/2e/f9/29b0f0d7764423573d35db4970dd573b324f4d41abe74d48adca542bcf79/rapidfuzz-3.14.6-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:28e9ce91bd41a8203185887ef9b1541a891aa61c5c1cb2e46f1689cd4288d372", size = 2401073, upload-time = "2026-08-30T21:43:08.742Z" }, + { url = "https://files.pythonhosted.org/packages/7a/f7/86ac824a7dd2b58729187cc31edebfa7805418f66d97d625010b7383d1de/rapidfuzz-3.14.6-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:864658e5a10d249a2277374e800f944fe990346d70eea6f3a51b712b6dd01984", size = 2786567, upload-time = "2026-08-30T21:43:11.048Z" }, + { url = "https://files.pythonhosted.org/packages/c6/a6/39fc42e45eb8ee70304862523b2e55cfbd2561c560dd8da1071015fa0ff0/rapidfuzz-3.14.6-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:3c2444f5cd757ded2c3ba8b1734253b801b9b2ba9ecb3ee40cd505cebbfa7341", size = 2504907, upload-time = "2026-08-30T21:43:13.281Z" }, + { url = "https://files.pythonhosted.org/packages/0a/ea/61f25272239ffef036eb3de1cc63372dfbff27193ca6f9f259d844f41a9c/rapidfuzz-3.14.6-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:2cc9b5dde0ac89f7856f997ef917cac8e18e9dea473e9b3090a84bd600de6a91", size = 3298728, upload-time = "2026-08-30T21:43:15.518Z" }, + { url = "https://files.pythonhosted.org/packages/6d/02/f9bfff9e19e852b097afa837a8000592bcd714fe80827a76367b958771b8/rapidfuzz-3.14.6-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:faebff9b9a287fb673f9a66465a7e03043601c9bfe5e71c3f91b3f2e7b8a37f6", size = 4272030, upload-time = "2026-08-30T21:43:17.785Z" }, + { url = "https://files.pythonhosted.org/packages/b3/d4/5845698661cb23bc7935536c28f5b86b2b3606de1f54722c1cfac39f170a/rapidfuzz-3.14.6-cp313-cp313-win32.whl", hash = "sha256:4406b2517b85febcf9419f8fbcdfbd534872ea32608050f9562224933ca49a4c", size = 1886313, upload-time = "2026-08-30T21:43:20.173Z" }, + { url = "https://files.pythonhosted.org/packages/67/f1/5b7c56737b9e5af7523ea79e90df732e9e4b2fa66fe2b333ee013ea6e541/rapidfuzz-3.14.6-cp313-cp313-win_amd64.whl", hash = "sha256:c69fb0e064d10c79908dcda76d7ca8ecdf8393a39acbb74dbad3f709f2c60e95", size = 1728638, upload-time = "2026-08-30T21:43:22.169Z" }, + { url = "https://files.pythonhosted.org/packages/05/5e/fc1da16b7f5245a7cc61dc08f70391ddaa1c538be1cf92681e7c763b77a4/rapidfuzz-3.14.6-cp313-cp313-win_arm64.whl", hash = "sha256:a0c8bef04f6b1d9fdbb319576350af53151a64692d477db7d4844c220bc8e212", size = 1185777, upload-time = "2026-08-30T21:43:24.27Z" }, + { url = "https://files.pythonhosted.org/packages/67/9e/8f862d2c8d80ee02633f1c9ce3e5121ce955e61efae24a61a05dd8a55fef/rapidfuzz-3.14.6-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:0f8d6718e7edacdb16455c0472e7552fd518decb91e91250c58784fd6163f54f", size = 1964420, upload-time = "2026-08-30T21:43:26.328Z" }, + { url = "https://files.pythonhosted.org/packages/3e/28/282e8c76b7dcc91e8f5aa1a594168d2136639f29dfda11384c6d36aabca0/rapidfuzz-3.14.6-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:8fa7d45388dec34a86038f2a38380f4922b74b5dd8991247f629a531178db10f", size = 1246072, upload-time = "2026-08-30T21:43:28.475Z" }, + { url = "https://files.pythonhosted.org/packages/4b/ae/8e0f714c55180667d66346e46a3d680dd9809bcee1c5f03557a58b4f2ef6/rapidfuzz-3.14.6-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:760ee152af5e8b4d241a469f933ba2d7215248618ae19770fec7d80d9e149db6", size = 1381829, upload-time = "2026-08-30T21:43:30.67Z" }, + { url = "https://files.pythonhosted.org/packages/eb/9a/4a106d68033a81c24ab71129e3016cc6a27a668f30f436e729cae79048e5/rapidfuzz-3.14.6-cp314-cp314-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:dbe3378db3ae0453accf6196e2ed943f43d416cfacdcb8883db105bc14a0130f", size = 1676195, upload-time = "2026-08-30T21:43:32.862Z" }, + { url = "https://files.pythonhosted.org/packages/6e/f0/b456a74d8e33051b76b3f156cf4d55f717614d68b44b6312ae1f5d85b31d/rapidfuzz-3.14.6-cp314-cp314-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:9ddb0ddf3ee616fdc066add4ef05639c5cf59b58d83779b6023488e5435f6191", size = 2714364, upload-time = "2026-08-30T21:43:35.003Z" }, + { url = "https://files.pythonhosted.org/packages/6d/56/1203b46cedefc3f0c16e10d87123fdd4ec0f2e209f65cd2bf221ec669217/rapidfuzz-3.14.6-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:08bc63b88048376114d1e66cf8fa6926495d03bb873eb87854fa74cf6848a70b", size = 3167618, upload-time = "2026-08-30T21:43:37.625Z" }, + { url = "https://files.pythonhosted.org/packages/57/17/fa4a0853979b885ff27488d9b80e7c5c985dfed74c5021ea95a3b54ddfad/rapidfuzz-3.14.6-cp314-cp314-manylinux_2_39_riscv64.whl", hash = "sha256:50cd6718bcda7ec5293635a9d0b3fb5906251013d3b99ca403ba9dfa8965f661", size = 1471360, upload-time = "2026-08-30T21:43:39.852Z" }, + { url = "https://files.pythonhosted.org/packages/7d/f2/757615ab88f7922b4477f9c93356c4512d744ea042e3e2b41554aab5ec1e/rapidfuzz-3.14.6-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:63b0e84faec3c5706cae8ae51246ff103407d54efa32a615a548b7b67392ebcf", size = 2403946, upload-time = "2026-08-30T21:43:42.038Z" }, + { url = "https://files.pythonhosted.org/packages/8f/c3/1c2670ff528f7e625d7b552e7ebccd5c4dfdcb84dc08ee85d1bcc0cf1465/rapidfuzz-3.14.6-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:9080a730fdcf3cb8a07464c90f9cf40c1b4ffc73a8375b56a8898aba619dda30", size = 2793123, upload-time = "2026-08-30T21:43:44.438Z" }, + { url = "https://files.pythonhosted.org/packages/5d/92/a01444687bb9a5a2679aa71325c227760e9c475cd02054b45fd8b219cb0c/rapidfuzz-3.14.6-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:178557c7a50c8c8d65369ede7f3d845bf23590a951c9a368caf166b105d58cf3", size = 2507361, upload-time = "2026-08-30T21:43:46.568Z" }, + { url = "https://files.pythonhosted.org/packages/98/90/43d80ba73fd297c744f7fe0a949af2a610b4b9be96688799c3e73d002b13/rapidfuzz-3.14.6-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:44f1cddbc2010700e2d88063d0ab64183efe2578d9b52770ce1cd283dda230c5", size = 3304287, upload-time = "2026-08-30T21:43:48.966Z" }, + { url = "https://files.pythonhosted.org/packages/5d/e9/fd9a160699b72b6857551642fe109a1d0a86b06b7ecc0d2b4bbecbc6b61b/rapidfuzz-3.14.6-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:17081a0e904c12bb4ed49619a2bbb6528f6af00fe850e7ace22487bfd2aea455", size = 4273338, upload-time = "2026-08-30T21:43:51.574Z" }, + { url = "https://files.pythonhosted.org/packages/d0/72/3bc42217fadd07ea0ff9d249cc8001d6f285197c253db95d3a03aac8c254/rapidfuzz-3.14.6-cp314-cp314-win32.whl", hash = "sha256:9e00c8c9500aacbc0c52b66369f54533ecbdcb92e5aa87e160fc8e293000a696", size = 1927357, upload-time = "2026-08-30T21:43:53.851Z" }, + { url = "https://files.pythonhosted.org/packages/57/8d/3ea3bf93a2f22858e1b1298126db35cbf58592d05571ca757f2f16071b17/rapidfuzz-3.14.6-cp314-cp314-win_amd64.whl", hash = "sha256:41ee893c4d7d0fb1844f6cad966540a833784b3bad2c239a0d80195d9231cef4", size = 1783090, upload-time = "2026-08-30T21:43:56.202Z" }, + { url = "https://files.pythonhosted.org/packages/13/17/4add9d94236b37b6f857a3bf34d696b32304e3debc6830584fda95413ac6/rapidfuzz-3.14.6-cp314-cp314-win_arm64.whl", hash = "sha256:10576c39fe6a49fad0bf1069371a77300ce166a3f36d2900d2d0bae08f297104", size = 1221915, upload-time = "2026-08-30T21:43:58.335Z" }, + { url = "https://files.pythonhosted.org/packages/23/a4/af0509bffac37645841e2a6b55a4c6c46f7b2fc0757610b0cba0cbcfa900/rapidfuzz-3.14.6-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:1b0a9546a7328d3cfc2f1385501db7c4c374fb566dc1a3b22ad56092846c0134", size = 1994141, upload-time = "2026-08-30T21:44:00.931Z" }, + { url = "https://files.pythonhosted.org/packages/67/da/d46da45e393937509111d4affa4db794fb064341735cfdcffe1f5f13a78a/rapidfuzz-3.14.6-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:9989280902b9c4ecf7de95fbb906e94df0d8c047290ed315c7aa1760cec9b3de", size = 1279969, upload-time = "2026-08-30T21:44:03.253Z" }, + { url = "https://files.pythonhosted.org/packages/4a/8a/1db5582d5c9684c57b1e292dc88d70177233b570e684fe30736140697658/rapidfuzz-3.14.6-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fc166efa4ca2fc9cc52e43784a54cbea95fc0e03e533f8266ef66b1c04c7cb76", size = 1381099, upload-time = "2026-08-30T21:44:05.402Z" }, + { url = "https://files.pythonhosted.org/packages/06/9b/a9dba69d174b4436c115fcd877a67745d355a859109e0f59955c14577519/rapidfuzz-3.14.6-cp314-cp314t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:32352a3ed1aad9c097d31fd4f2eece3030169e2de3dedde7a2fadc2652b768ad", size = 1638869, upload-time = "2026-08-30T21:44:07.51Z" }, + { url = "https://files.pythonhosted.org/packages/61/34/67915218f5f84ec2cda57560d81425929b8ea97956eb31283bf95768fefc/rapidfuzz-3.14.6-cp314-cp314t-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ecb45d616002751b58914d5b7c2e66acd39e12242be12717a1258148a1b36526", size = 2687831, upload-time = "2026-08-30T21:44:09.709Z" }, + { url = "https://files.pythonhosted.org/packages/5e/80/07985e10b534dbdd48df0ddf2e42f9d27cf98dc44e09fe047fc4b38471f5/rapidfuzz-3.14.6-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6f9ad513e3a3e045b60b421d5cd3887ae0a33b38fc6c6db3ea5e27c0a2e0412c", size = 3185373, upload-time = "2026-08-30T21:44:12.162Z" }, + { url = "https://files.pythonhosted.org/packages/91/09/db64291ce5f11c0f79486b435b49f5dc66680f605077cb011d282bf767b4/rapidfuzz-3.14.6-cp314-cp314t-manylinux_2_39_riscv64.whl", hash = "sha256:f35723caef8cc31b6f34209708fb172fc88bab0077c12e9b36bbb829baaf1b16", size = 1459628, upload-time = "2026-08-30T21:44:14.427Z" }, + { url = "https://files.pythonhosted.org/packages/d0/99/7eeaf6f7f42d4ec8b90db54c73f7c2a727e208b4db6fd5ea807e87133b9c/rapidfuzz-3.14.6-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:408b2e8e8c1ac71b57f0923cf964d6932539725e07b69e70ec66f22c4a403891", size = 2407348, upload-time = "2026-08-30T21:44:16.832Z" }, + { url = "https://files.pythonhosted.org/packages/19/bb/db04caff7bf26718e97592f8cc007988ef18eb088ebb0742addcb25f0819/rapidfuzz-3.14.6-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:5667c56fdc902fa1e12449b5c042e8b1c7e9b30040db20c396fbdb3d0a750866", size = 2758630, upload-time = "2026-08-30T21:44:19.196Z" }, + { url = "https://files.pythonhosted.org/packages/3f/26/962fc396a56ec37146eb5331e55ae53d19dc564fd921f49a6d524c83ee05/rapidfuzz-3.14.6-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:76a122fc573df603deb5fb827df31bb5efbd0826b50bb7aeca8535a6e8c70cf9", size = 2494519, upload-time = "2026-08-30T21:44:21.687Z" }, + { url = "https://files.pythonhosted.org/packages/83/0f/d2067e23d9b7fb2aeb70a6b36173f0b2376635483f670aa5c47f17e55135/rapidfuzz-3.14.6-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:e221366e24709b9d41d5f9cc99053b04cfc575d429e956a82cfbc4c4e9e8860a", size = 3262241, upload-time = "2026-08-30T21:44:24.218Z" }, + { url = "https://files.pythonhosted.org/packages/ce/bd/05e48e21b1dd722b41c0cb8ab8867996f6e0c0a1b46e42921ace09799b0c/rapidfuzz-3.14.6-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:36710ff214b7a8049d26a9c81d99948026593cacb47663742c4119072b651ecd", size = 4296246, upload-time = "2026-08-30T21:44:26.911Z" }, + { url = "https://files.pythonhosted.org/packages/12/ce/f4b355f05b17bdb3a56f1c5e9bd864965dbb810f93d1b5d6044ecfcbd42d/rapidfuzz-3.14.6-cp314-cp314t-win32.whl", hash = "sha256:66ece6f5e2586c742fc3e0b8487e06783d27c6c24adcdcfdd7f306afbd8b5737", size = 1977694, upload-time = "2026-08-30T21:44:29.431Z" }, + { url = "https://files.pythonhosted.org/packages/4a/15/d2c20c57b357ec4157e74a197b3f622dbda0b2a82d1fc708ed7b262758f9/rapidfuzz-3.14.6-cp314-cp314t-win_amd64.whl", hash = "sha256:cab4a932cec02d09471e2c9f1434049ef5bfe1f6e646ff10939c222dc610ad60", size = 1827262, upload-time = "2026-08-30T21:44:31.683Z" }, + { url = "https://files.pythonhosted.org/packages/15/e5/c38c19fbc1de82980e05bd3adbe1dc7f3dd0680e38e868646082317572d6/rapidfuzz-3.14.6-cp314-cp314t-win_arm64.whl", hash = "sha256:b056ce19eaea2ea70c6a6fb387a605ca2af8979de5b9d507597e8012820ddb14", size = 1245604, upload-time = "2026-08-30T21:44:34.066Z" }, + { url = "https://files.pythonhosted.org/packages/10/37/b015bf56f88e9b18b81ad462f610e70cc1145a9df39154fcbe7ddf9f8868/rapidfuzz-3.14.6-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:bc3d74d18543ddfbc8babe1faadb19927a7999fd0d01181cce9e721c14c36ab6", size = 1964451, upload-time = "2026-08-30T21:44:36.695Z" }, + { url = "https://files.pythonhosted.org/packages/d2/1a/7b88284d85b4f7dfdf3038263e11eb11871472aa32902c7063a5fdd7a7c5/rapidfuzz-3.14.6-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:aaa83b633d877a05d549d2073629134998d1b3b9dbc114873d3ff4277984979f", size = 1245701, upload-time = "2026-08-30T21:44:38.841Z" }, + { url = "https://files.pythonhosted.org/packages/a7/f3/444d939f4b6c3c86f67083cb792978f3f42c28f944e66e9152e910cd212a/rapidfuzz-3.14.6-cp315-cp315-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cbe6a62f71fcbca72acbf5a30e53380600369f257f951d664d81d30c0c598595", size = 1382770, upload-time = "2026-08-30T21:44:40.978Z" }, + { url = "https://files.pythonhosted.org/packages/23/a8/1830f07f7d3fcc56508135f130dbd24a917ddedb71107b04b2fbb33d5da9/rapidfuzz-3.14.6-cp315-cp315-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b82c21c30568e096ef2a9dda7d45c379e6141694e0472dac73bc4372ce13ccee", size = 3163591, upload-time = "2026-08-30T21:44:43.513Z" }, + { url = "https://files.pythonhosted.org/packages/10/e8/da76d94af820707dcbfce224b635fb7c389c19525426c31645c97bedd601/rapidfuzz-3.14.6-cp315-cp315-manylinux_2_39_riscv64.whl", hash = "sha256:fc950bb77105a2717d03d9f9c9e21e9ace7df2b8e864dd91edef7e32fa143be2", size = 1467985, upload-time = "2026-08-30T21:44:45.871Z" }, + { url = "https://files.pythonhosted.org/packages/30/75/5cfc0d1491e3c60a8669e8e2b78942c4f395cccabfb9c73bc8b209664e29/rapidfuzz-3.14.6-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:c53a269bdbd71ffbc856d3db9e609478251001ee272507578fa838bc2bd421fe", size = 2405269, upload-time = "2026-08-30T21:44:48.376Z" }, + { url = "https://files.pythonhosted.org/packages/c3/81/9c522c26cfe1909714eb840856106f1e419a44c4e0de034a3eeb873da00b/rapidfuzz-3.14.6-cp315-cp315-musllinux_1_2_riscv64.whl", hash = "sha256:bf4fb0f19c9dfce7a908c3e309753602ce3edb83bb74e9ff997e278765bf89df", size = 2507334, upload-time = "2026-08-30T21:44:50.903Z" }, + { url = "https://files.pythonhosted.org/packages/40/29/0bbd158eeddf05e5b581f89bf7c9f0cf330953579309b3806862d360a454/rapidfuzz-3.14.6-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:189ce2bf14938bfa003fbbe7e6da7584ed6ebbc4c560686255dbc20e2829f470", size = 4270388, upload-time = "2026-08-30T21:44:55.271Z" }, + { url = "https://files.pythonhosted.org/packages/be/be/2b67b32988cb96b7fa9461ff3436e275716df00f7817212ed0a1c1779062/rapidfuzz-3.14.6-cp315-cp315-win32.whl", hash = "sha256:7ca0f498bf771a87557e6d8b573aa6cf3daded58ae2eaeb6973618ce3e1615ad", size = 1927539, upload-time = "2026-08-30T21:44:57.796Z" }, + { url = "https://files.pythonhosted.org/packages/51/42/640e1bd16422392fbb6394def1f7dfd4d05bd13c986016ce4b3f91295430/rapidfuzz-3.14.6-cp315-cp315-win_amd64.whl", hash = "sha256:d4c5adb921b67dd79ffc0a14f92b9f8df3d012e66aab340b154ed87014229d93", size = 1783415, upload-time = "2026-08-30T21:45:00.091Z" }, + { url = "https://files.pythonhosted.org/packages/06/ba/c6966904eb7b3d1c6344e6c29245447625d156b11e9757b29adc3cb46037/rapidfuzz-3.14.6-cp315-cp315-win_arm64.whl", hash = "sha256:c9d135fb93709d707577da8a7a8ffc7283525a5b6d0ce55aa3724be5639ed65b", size = 1221931, upload-time = "2026-08-30T21:45:02.531Z" }, + { url = "https://files.pythonhosted.org/packages/ae/97/6dd7f10756eb703e11803c5c838191c2151112f632e29f5eacb1ed1cf86c/rapidfuzz-3.14.6-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:dd89abd1c4b3776c3471a817216830bd275441c8344bbda5d51a3bffe1e0fbdf", size = 1985107, upload-time = "2026-08-30T21:45:04.965Z" }, + { url = "https://files.pythonhosted.org/packages/75/4a/be587adefd9539a89cc6016bac44d222cda4c8212856759c82501fd89e4a/rapidfuzz-3.14.6-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:eab2d4680d7f438dbb1d484b187d59a943edea9c83f792c764a0c148a417a60a", size = 1272093, upload-time = "2026-08-30T21:45:07.304Z" }, + { url = "https://files.pythonhosted.org/packages/de/3f/982b2f1b2a16c46d4598829b6b2d7185921f146d5893630f917cb9d27542/rapidfuzz-3.14.6-cp315-cp315t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8683fefdd3484d64a191b3efbc8cbe9162c3eac891fd62d0a1b70e117ffcd434", size = 1371112, upload-time = "2026-08-30T21:45:09.699Z" }, + { url = "https://files.pythonhosted.org/packages/e6/12/2a1fe61cb9f0ac0dc4166bcb016df695047e75251481a197d47aa5ce8ea5/rapidfuzz-3.14.6-cp315-cp315t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2bc7af3a699371a941aac86dc8a79ac92adeb3c2add2aab02230e76068a0029e", size = 3175780, upload-time = "2026-08-30T21:45:12.266Z" }, + { url = "https://files.pythonhosted.org/packages/8d/01/abd33d0b7595643e598802a07466af388f1560d7b7cb70f442cc292f4067/rapidfuzz-3.14.6-cp315-cp315t-manylinux_2_39_riscv64.whl", hash = "sha256:40c2753e2d4dc96b25f8a25adc23ab0bb6cfd8bc8125a1753ac4b037d6ff6511", size = 1458364, upload-time = "2026-08-30T21:45:14.68Z" }, + { url = "https://files.pythonhosted.org/packages/9d/8e/efc98b0cfb540f41661f6a8bf21b67807e221102e5e8fb1585233b39a3bd/rapidfuzz-3.14.6-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:36a37ddc729c33618d89fa221d3333b9b956dc38cf15d31301e6169d962399a3", size = 2398037, upload-time = "2026-08-30T21:45:17.434Z" }, + { url = "https://files.pythonhosted.org/packages/7f/c1/4d89214a453215d897cc76cd6e13937c8ea5dc9f8217993fe2b1eeaf39a5/rapidfuzz-3.14.6-cp315-cp315t-musllinux_1_2_riscv64.whl", hash = "sha256:635f242f4bdf05d1477fa409815bd73e5f78896773ace84997bc472ffeef685f", size = 2497044, upload-time = "2026-08-30T21:45:20.328Z" }, + { url = "https://files.pythonhosted.org/packages/a9/2d/70aacf6cb577470bdd6f06890d25ecb7ee8a56baa07b114d5877a93ecedd/rapidfuzz-3.14.6-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:40d0cd9c82083aeb30bae8dee265ae571e6748d0d7b222ddd777f33d95a3b712", size = 4284741, upload-time = "2026-08-30T21:45:22.983Z" }, + { url = "https://files.pythonhosted.org/packages/92/7d/943a04a134a5d333c00d3a77169226defef5e081be9219a765afc176dda0/rapidfuzz-3.14.6-cp315-cp315t-win32.whl", hash = "sha256:15da2b258908eb38853c1a6a58a1d09d9aad9c721e03a68c8ba691cd31dff739", size = 1974478, upload-time = "2026-08-30T21:45:25.475Z" }, + { url = "https://files.pythonhosted.org/packages/21/0e/8356ca3e190e2bcced9b80e374d95b0925c4716b51e65720a55399983f41/rapidfuzz-3.14.6-cp315-cp315t-win_amd64.whl", hash = "sha256:3d502769263318690d4f6638b08483979d1b88cdc7c6f087482eea935fde4031", size = 1823286, upload-time = "2026-08-30T21:45:28.368Z" }, + { url = "https://files.pythonhosted.org/packages/fb/04/a0b0e6324b6384d1ab40feb4d16400af3b3101d38cbd15957edd9d17cbe0/rapidfuzz-3.14.6-cp315-cp315t-win_arm64.whl", hash = "sha256:07c7aa0b1e4b9999a54f9e73317d6743ff85442c8ef7b7fbbe6b190fd37d9e75", size = 1243815, upload-time = "2026-08-30T21:45:31.187Z" }, + { url = "https://files.pythonhosted.org/packages/08/9a/7d4949406e2d391e160ead12036bba05e7c90e09bba77a782d33e7e6a1b0/rapidfuzz-3.14.6-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:0844066900cdc9909ce4ab4fb5ba1d8e0c021252d770f2ea476f3443df1d22ef", size = 1912210, upload-time = "2026-08-30T21:45:33.653Z" }, + { url = "https://files.pythonhosted.org/packages/7c/00/a1a077f5cf90c9fa13b28c721f931529ad02748d418d7750590a388832a9/rapidfuzz-3.14.6-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:1398bd2c197b79bfc40b615999fd3599dc60265fdd5b59edc18156ae048c4cde", size = 1209219, upload-time = "2026-08-30T21:45:36.035Z" }, + { url = "https://files.pythonhosted.org/packages/48/69/a573c2e5e1b1a4f19e98a8fb3f6a792a44f5b8a067895a2654890ffd35a4/rapidfuzz-3.14.6-pp311-pypy311_pp73-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e2fc748d1fde4109e5d0dab27f1e61f53b3136a235dfee5a4fb579da44808b6a", size = 1361237, upload-time = "2026-08-30T21:45:38.582Z" }, + { url = "https://files.pythonhosted.org/packages/ea/0b/375ebdfc4ca149e23793bb6b72461954ec64d0acbb826030787e88b90ff3/rapidfuzz-3.14.6-pp311-pypy311_pp73-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b42536675c930cb76b7998bfc4d8e59cb35d8df47f2103020265743b6b2ccd2a", size = 3136631, upload-time = "2026-08-30T21:45:41.426Z" }, + { url = "https://files.pythonhosted.org/packages/55/56/799accc99532ecaaa2c1d04c7e594d6bb8f1afdddc327389c61196741cb8/rapidfuzz-3.14.6-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:1e6911e3a14971719ddc35af98f181d2e5369ab273a5a3488ab7685d23c31ad5", size = 1722739, upload-time = "2026-08-30T21:45:44.301Z" }, +] + [[package]] name = "requests" version = "2.32.5"