Repository navigation
feat(upgrade): Update to use latest version, implement optimizations - #8
Draft
abimaelmartell wants to merge 14 commits into
Draft
abimaelmartell wants to merge 14 commits into
abimaelmartell wants to merge 14 commits into
Conversation
First requests were ~80s vs ~13s on repeat due to CUDA JIT-compiling GPU kernels for new tensor shapes (especially OCR-det). This adds a warm-up pass at startup that runs full pipeline inference on a synthetic 4-page PDF, forcing kernel compilation before real traffic arrives. Also fixes warm-up not running at all in debug server mode. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… VRAM - Move synthetic PDF generation into app/warmup.py with 8 layout strategies (title+body, two-column, scattered snippets, alternating blocks, grid, isolated words, dense text, varied font sizes) to produce 30-40+ unique OCR-det resolution groups instead of the previous 7 from uniform text pages. - Remove hardcoded MINERU_VIRTUAL_VRAM_SIZE=24 from Dockerfile so the actual GPU VRAM is auto-detected (A40 has 45GB, was capped at 24GB). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
A40 with 45GB was using the same batch_ratio=32 as 24GB cards. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The synthetic PDF warmup produced resolution groups with zero overlap to real documents (28 vs 37 shapes, none matching). Add direct tensor warmup that feeds dummy images at every 64px-padded resolution (160 shapes) into the OCR-det model, guaranteeing CUDA kernels are pre-compiled regardless of document content. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two fixes for the OCR-det cold-start penalty: 1. Set torch.backends.cudnn.benchmark=False — eliminates the ~2s per-shape auto-tuning that cuDNN runs for each new input resolution. Drops per-shape overhead from ~2s to ~0.05s. This is the primary fix. 2. Fix warmup shapes to match actual CUDA tensor dimensions. The previous grid (heights 64-512, stride 64) missed all shapes at heights 544-960 produced by DetResizeForTest (limit_side_len=960, round to 32px). Now simulates the full preprocessing transform on realistic crop sizes to compute the exact ~255 unique CUDA shapes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
cuDNN/CUDA caches compiled kernels per tensor shape including the batch dimension (N). Warmup compiles shapes at N=1, but real inference uses N=group_size (variable). This mismatch caused ~17 of 37 shapes to miss the cache on first request (~2s each = ~34s overhead). Changes: - Add batch_analyze_det_bs.patch: caps OCR-det max_batch_size to 1 via OCR_DET_MAX_FORWARD_BATCH env var (default=1, set 0 to disable) - Add batch size diagnostic to warmup: tests N=1 vs N=4 timing to confirm whether batch dimension is part of the cache key - Update warmup docstring to note 200 DPI and thread requirement Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split PDFs into chunks and process concurrently on N GPU threads (default 3). CPU pre/post-processing from one chunk overlaps with GPU work from another, improving utilization on the A40. Expected ~2x speedup for 9-page PDFs and enables 100-page PDFs to complete within timeout. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
pypdf's PdfWriter produces chunks with broken xref tables that pypdfium2 can't parse. Round-trip each chunk through _repair_pdf during splitting instead of failing and retrying every time. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
pypdfium2 document loading isn't thread-safe — concurrent loads cause PdfiumError. Stagger chunk submissions by 1s so loading phases don't overlap while inference still runs in parallel via GIL-releasing CUDA ops. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upgrades MinerU to 2.7.6 and addresses a critical performance issue where the first request to process a PDF took ~80s while identical repeat requests took ~13s. The root cause was CUDA JIT-compiling GPU kernels on first encounter with new tensor shapes, particularly in the OCR detection step (37 resolution groups at 1.92s each vs 50 items/s once cached).
This PR adds a synthetic inference warm-up at startup that generates a 4-page text-dense PDF and runs the full pipeline in OCR mode, forcing kernel compilation before real traffic hits. It also adds auto-repair for broken PDFs via pypdf, returns processing metadata in responses, enables GPU batch optimizations (batch_ratio=32 for 24GB VRAM), and fixes warm-up not running at all in debug server mode.
Key changes
app/serverless.py— Refactored conversion into retryable _do_convert(), added PDF repair, synthetic warm-up PDF generation, and moved model warm-up to run in both debug and production modesDockerfile— Enable batch patch, set MINERU_VIRTUAL_VRAM_SIZE=24, MINERU_PDF_RENDER_THREADS=8, OMP_NUM_THREADS=8patch/mineru_batch.patch— Updated for MinerU 2.7.6, adds 32x batch tier, forces OCR det batching via env var