Repository navigation
fix: gate low-quality PDF extraction before OCR fallback - #105
Merged
Merged
Conversation
… gates Red against 21564da. Real-binary fixtures (skipped without pdf-extract on PATH): tabular text_layer pages keep their digits, one text page plus nine scans OCRs only the scans. Fake-binary cases: a gate rejection with OCR missing or empty keeps the direct text and hints once, garbage plain text without verdicts still falls back to OCR, and `docs ocr` honours configured thresholds. Existing gate tests now expect a flagged result, not None. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wdqj51cPquyTRKmoJLQV6y
The whole-document gates returned None for any failure, so ingest OCR'd every page: tabular FOIA logs (alpha ratio well under 0.20) lost their exact text-layer digits, one bad page sent a whole document to OCR, and without tesseract the document was stored with no text while the ImportError was swallowed. - extract_text: with per-page verdicts only min_chars applies (verdicts already route garbled/image_only pages one by one). Plain output that fails the per-page or alpha gates comes back with quality_reasons set instead of None. - ingest: gated text goes to whole-document OCR; if OCR is missing, fails, or finds nothing, the direct text is kept and the install-extras hint prints once. - OCREngine.process_pdf (docs ocr): unset pdf_extract_* settings come from config instead of hardcoded defaults, and gated text is kept when OCR is unavailable or empty. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wdqj51cPquyTRKmoJLQV6y
Runs the real `pdf-extract --json` over a directory and reports, per file and per top-level folder, text-layer chars kept and pages sent to OCR under main's policy, #105 as submitted, and the fixed gates. Stdlib only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wdqj51cPquyTRKmoJLQV6y
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wdqj51cPquyTRKmoJLQV6y
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #61
Summary
Verification
pytest tests/test_pdf_extract_json.py tests/test_document_pages.py -q(37 passed)ruff check openfoia/config.py openfoia/pipeline/pdf_extract.py openfoia/pipeline/ingest.py tests/test_pdf_extract_json.py tests/test_document_pages.pyruff format --check openfoia/config.py openfoia/pipeline/pdf_extract.py openfoia/pipeline/ingest.py tests/test_pdf_extract_json.py tests/test_document_pages.pyEvidence
Follow-up commits (Claude): e4c8959 failing tests, 43681b3 fix, cbb7902 bench script, 6f5b857 changelog.
What changed after review. The whole-document gates used to return
None, which sent the whole file to OCR. Now they only judge plain output from an olderpdf-extractwith no per-page verdicts. With verdicts, garbled and image_only pages still go to OCR one at a time, the same as on main. Text that fails a gate is still sent to OCR, but if tesseract is missing or OCR returns nothing, the direct text is kept and theinstall-extras ocrhint prints once.docs ocrnow reads the configured thresholds.Real corpus (180 PDFs, 16,603 pages,
pdf-extract0.7.0; 1 file fails to extract under every policy). Kept = non-whitespace text-layer chars kept; OCR = pages sent to OCR.files where the policy differs from main: pr 0, fixed 0
not extracted: exit 1 1
On this corpus none of the three policies differ: every file is either over the gates or already routed per page by its verdicts. So the corpus shows no gain from the gates, and no harm from them either. The harm shows up on tabular FOIA responses, which these hand-written fixtures reproduce:
As submitted, the PR OCR'd a FOIA log (alpha ratio 0.07) and a budget table (0.01), throwing away 2,889 exact text-layer chars. It also sent a 1-text-page + 9-scan file to OCR in full, all 10 pages. The fix matches main on all five.
Tests: 500 passed before, 508 passed after (
pytest tests/ -q --deselect tests/test_extraction.py, 20 skipped). The fixture tests run through the real binary whenpdf-extractis on PATH and skip cleanly when it is not.