Multimodal Pipelines: OCR, Vision, and Audio in One Stack

One big vision-language model does not replace a document pipeline. The architecture that survives production is: classify each input, route it to a specialist per modality (a text extractor, an OCR engine, a VLM, an ASR model), normalise everything into a single JSON shape with confidence and provenance, and only then let an LLM near it. Feeding raw PDFs to a multimodal API works in demos and fails in production on three axes at once: cost (you're paying vision-token prices for born-digital text), reliability (page 47 of 200 silently hallucinated), and auditability (no way to trace a wrong answer back to a source region).

Preprocessing decides your ceiling

Most "OCR quality" problems are routing problems. A PDF is two different inputs wearing the same extension: born-digital (there's a real text layer — extract it with PyMuPDF or pdftotext, which is free, instant, and character-perfect) and scanned (rasterise at 300 DPI, deskew, then OCR). The router is ten lines: if the text layer covers most of the page area and isn't gibberish, skip OCR entirely. On a typical business-document mix, 60–80% of pages take the free path, which is also why per-page cost estimates that assume OCR everywhere are wrong by 3–5×.

Audio gets the equivalent treatment: run voice-activity detection (silero-vad) first so you never pay to transcribe silence, then chunk to ~30s segments with overlap for the ASR pass.

Pick the OCR engine per document, not per benchmark

The load-bearing distinction is in the last column. Classical OCR fails loudly — you get £$%^ and a low confidence score, and you can route the page to a human. VLMs fail plausibly: they will "read" a smudged total as a clean number, invoice you never wrote, with no confidence signal attached. For anything where a wrong digit matters, classical OCR with confidence scores plus human review of low-confidence fields beats a better-on-average VLM. I use VLMs as the fallback tier for pages Tesseract scores badly, not as the default — you get VLM quality on the 15% of pages that need it at 15% of the price. The Tesseract docs are still the reference for tuning the classical tier (PSM modes alone fix half of "Tesseract is bad" complaints).

Audio: faster-whisper, then the diarisation gap

For transcription, faster-whisper running large-v3 at int8 is the default answer: roughly 10× real time on a single mid-range GPU, meaning an hour of audio in about six minutes, with word-level timestamps. On CPU, small or distil variants keep it usable at 1–2× real time. The part Whisper doesn't do is diarisation — who spoke — which is a separate pass (pyannote is the standard) that you align to the transcript by timestamp. Budget for diarisation being the flakiest stage in the whole pipeline; overlapping speakers and cross-talk degrade it well before the transcription itself suffers.

Fuse to one shape, with provenance

Every extractor's output normalises to the same record before anything downstream sees it:

{
  "source": "contracts/msa-2025-11.pdf",
  "modality": "pdf-scan",
  "extractor": "tesseract-5.4",
  "page": 12,
  "spans": [
    {
      "text": "Termination requires 90 days written notice.",
      "bbox": [88, 412, 530, 436],
      "confidence": 0.94
    }
  ]
}

Audio spans carry t0/t1 seconds instead of bbox. This one decision buys you three things: chunking and embedding code that's modality-agnostic, extraction with structured outputs that can cite the span it pulled each field from, and — the one that matters in an argument — the ability to show a human the exact page region behind any answer. Store the extractor name and version in every record; when you upgrade an engine, you'll want to reprocess exactly the records the old one produced and nothing else.

Confidence thresholds and the human queue

Set thresholds per field, not per document. A contracts pipeline can accept 0.85 confidence on a paragraph of boilerplate and must not accept 0.99 on a payment amount without a checksum or a human glance — the cost of error is the variable, not the OCR score. In practice: numeric fields that feed decisions get validated (totals cross-checked against line items, dates parsed strictly), and validation failure routes the page image plus the extraction to a review queue. Expect 5–15% of real-world scanned pages to need the queue at first; the rate falls as you tune the classical tier.

This is the same architecture Paperless-ngx applies to home document management — OCR everything, keep the original image, make the text searchable but never authoritative — and it's worth studying as prior art even for commercial pipelines.

The LLM comes last, and locally if you can

By the time an LLM sees anything, it's seeing normalised text spans, not pixels. Field extraction, classification, and summarisation over those spans are exactly the tasks where an 8B model with constrained decoding performs within noise of frontier APIs, so a single 24GB GPU running Ollama can own the entire post-extraction stage — no document content leaves the building, which for contracts and medical intake is frequently the requirement that justified the pipeline in the first place.

What I'd do

Route first: text-layer extraction for born-digital pages, Tesseract with confidence scores for clean scans, a VLM only for the pages Tesseract flags, faster-whisper int8 for audio with silero-vad in front. Normalise everything to one span schema with extractor version and provenance, validate decision-feeding fields with rules, and queue failures for humans. Keep the LLM at the end of the pipe, local where the data is sensitive. It's five components instead of one API call, and it's the difference between a system you can debug page-by-page and a black box that's confidently wrong somewhere in the middle of a 200-page PDF.