The big idea
What is document AI?
Businesses drown in documents — invoices, forms, contracts, receipts, IDs — that humans manually key into systems. You want to turn a messy scanned invoice into clean structured data: {vendor, invoice_no, total, line_items[]}, automatically, at scale. But documents are visual, varied, and full of layout (tables, forms, columns) that a plain text dump throws away.
A document AI (intelligent document processing) pipeline turns unstructured documents into structured data: preprocess and OCR the image, understand its 2D layout, extract fields and tables (increasingly with a VLM reading the doc directly), validate with confidence scores, and route the doubtful cases to a human. Reading the pixels is easy; producing correct structured fields is the system.
How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the pipeline grow, and at the end flatten the layout and drop validation to see why 2D structure and human review matter. Hit Begin.
Step 1 · OCR alone isn't enough
Text is not structure
The naive approach: run OCR and return the text. OCR is great at reading characters. So why isn't "the text of the document" the same as the structured data you need?
Design decision: Run OCR and return the text. Why isn't that the structured data you need?
The call: OCR can't read printed text. — OCR reads printed text well — that's its job. The gap is structure: turning correctly-read characters into fields and tables requires understanding the page's 2D layout, which raw OCR text doesn't preserve.
OCR reads the characters, but flattens the page into a linear text stream, discarding the 2D structure that carries the meaning: which text is a label vs a value, which numbers belong to which table cell, what's a header. Structured data — fields and tables — lives in that spatial layout. So "the text" isn't "the data"; you need to understand the layout and extract structure, not just recognize characters.
Reading ≠ understanding: OCR is character recognition; document AI is document understanding. The hard, valuable part isn't reading "4471" — it's knowing that "4471" is the invoice number because of where it sits relative to its label. Layout is meaning.
Step 2 · A pipeline, not a step
Ingest → understand → extract → validate
Turning a document into trustworthy structured data is several distinct jobs. What are the stages, and what does each contribute?
Run a pipeline: ingest the document → preprocess + OCR (clean the image, read text + coordinates) → layout analysis (understand the 2D structure) → extraction (produce fields and tables) → validation (check + score confidence) → structured output, with low-confidence cases routed to a human. Each stage removes a different failure: OCR handles pixels, layout handles structure, extraction handles meaning, validation handles correctness.
Stages with distinct jobs: Document understanding decomposes into recognizing characters, understanding layout, extracting fields, and verifying them. Separating these lets each use the right technique and lets you measure/route by confidence — a monolith "read the doc" hides where errors come from.
Step 3 · Read the pixels
Preprocessing & OCR
Real documents are photographed, skewed, low-contrast, or noisy scans. Before understanding anything, you must reliably get the text off the page. What does this stage do?
Preprocess the image — deskew, denoise, adjust contrast, handle rotation and multi-page — then run OCR to extract not just the text but the bounding-box coordinates of every word/line. Those coordinates are crucial: they're what the next stages use to reconstruct layout. Good preprocessing lifts OCR accuracy (garbage pixels → garbage text), and keeping word positions is what lets "reading" become "understanding."
Text + positions: OCR here outputs located text — every token with its box on the page. That spatial information is the bridge from characters to structure: layout analysis and extraction both rely on where text is, not just what it says. Clean the image first, because OCR errors cascade.
Step 4 · Understand the page
Layout analysis
Now the pivotal step. You have located text, but a form, a table, and a paragraph all need to be read differently. How do you recover the document's 2D structure?
Design decision: You have text with coordinates. How do you recover the document's structure (forms, tables)?
The call: Use layout analysis — detect regions, tables and key-value pairs from the spatial arrangement (layout-aware models like LayoutLM, or a VLM that sees the image). — Layout analysis models the page in 2D: it detects blocks/regions, table cells (rows/columns), and the spatial links between field labels and their values. Layout-aware models (e.g. LayoutLM-style, combining text + position + image) or a VLM that sees the page directly recover the structure that linear text loses.
Layout analysis models the page in 2D: detect regions (header, blocks), tables (rows/columns/cells), and key-value relationships (which label owns which value) from the spatial arrangement. Use layout-aware models (LayoutLM-style: text + position + image together) or a VLM that sees the page image directly. This is the crux — it recovers exactly the structure OCR flattened, turning located text into an understood document.
Position is a feature: Modern doc AI treats a document as text plus 2D position plus image. Layout-aware transformers (LayoutLM and kin) fuse these so the model knows "this number is in the total row." A VLM sidesteps explicit OCR by reading the image — but the principle is the same: layout is signal, not noise.
Step 5 · Pull the fields
Extraction: templates vs ML vs VLM
Now produce the actual structured output — the fields and tables you want. Documents range from fixed forms to wildly varied invoices. What extraction approach fits?
Match the method to the variety. Templates/rules work for fixed layouts (a form that's always identical — extract by position/regex). ML models (layout-aware) handle varied documents by learning to tag fields. Increasingly, a VLM/LLM reads the document (image and/or OCR text) and outputs structured JSON directly to a schema — flexible across document types with little per-template work. Often you classify the document type first, then route to the right extractor.
Fixed → template, varied → learned: Templates are cheap and exact but brittle to layout changes; ML/VLM generalize across formats at the cost of needing validation (they can be wrong or hallucinate). Modern pipelines lean on VLMs prompted with a target schema, but keep classification + routing so each doc type gets the best extractor.
Step 6 · Is it right?
Validation & confidence
Extraction gives you values — but some are misread, ambiguous, or (with an LLM extractor) hallucinated. When this data feeds payments or records, a confidently-wrong field is dangerous. How do you know which values to trust?
Design decision: Extracted fields may be misread or hallucinated. How do you know which to trust?
The call: Trust the extractor — modern models are accurate. — Even accurate models misread bad scans and can hallucinate field values, and downstream use (payments/records) makes a wrong value costly. You must validate and score confidence, not blindly trust.
Validate every extracted value: check type/format (a date is a date, currency parses, an ID matches its pattern), apply business rules (line items sum to the total, required fields present), and cross-check across the document. Attach a confidence score to each field. Values that fail a check or score low are flagged, not trusted — this is what catches OCR misreads and, critically, LLM-hallucinated values before they reach payments, records, or contracts.
Never trust blindly: Validation converts "the model said X" into "X passes our checks and we're N% confident." Business-rule cross-checks (totals, patterns) are powerful because documents are internally consistent. Confidence scores are what make the human-in-the-loop routing (next) possible.
Step 7 · Humans on the hard cases
Human-in-the-loop
Some documents are genuinely hard — awful scans, handwriting, unusual layouts — and full automation would either error out or produce wrong data. You can't accept mistakes on critical records, but you also can't review everything by hand. How do you balance automation and accuracy?
Use confidence-based human-in-the-loop: auto-accept high-confidence extractions, and route low-confidence ones (or specific critical fields) to a human reviewer who corrects them. This keeps automation high while guaranteeing accuracy where it matters. Crucially, close the feedback loop: human corrections become labeled data that improves the models, so the share needing review shrinks over time. You automate the easy majority and spend human attention only where confidence is low.
Confidence gates the human: The confidence score is the dial: set a threshold, auto-process above it, review below it. This makes the system both scalable (most docs are automated) and trustworthy (humans catch the hard ones), and the corrections continuously retrain the models — a flywheel that raises the automation rate.
Step 8 · The sharp edges
Scans, tables, hallucination & PII
Real document processing hits hard corners: terrible scans and handwriting, endlessly varied layouts, tables spanning pages, LLM hallucination of field values, and sensitive PII/compliance.
Improve bad inputs with preprocessing and robust/handwriting-capable models, and lean on human review when quality is too low. Handle varied layouts with document classification + routing and layout-general models rather than one brittle template. Manage multi-page tables by stitching rows across pages. Guard against hallucination: constrain extractors to ground each field in the document (return the source location/box), and validate — never let a VLM invent a value with no evidence. Treat documents as sensitive: PII redaction, access control, retention, and audit trails for compliance. And run it as an async batch pipeline that scales across document types.
Design for the unhappy path: Bad scans → preprocess + robust models + HITL. Varied layouts → classify + route. Cross-page tables → stitch. Hallucination → ground fields in the doc + validate. PII → redact/control/audit. Document AI turns messy reality into clean data, so most of the engineering is handling how messy and how consequential that reality is.
You did it
You just designed a document AI pipeline.
- Plain OCR gives characters but flattens the 2D layout — text is not structured data.
- Run a pipeline: ingest → preprocess/OCR → layout → extraction → validation → structured output.
- Preprocess then OCR to get located text (words + bounding-box coordinates).
- Layout analysis recovers 2D structure (tables, key-value pairs) — the crux, via LayoutLM-style models or a VLM.
- Extract with templates (fixed forms), ML, or a VLM outputting schema JSON; classify + route by doc type.
- Validate against schema/rules and score confidence to catch misreads and hallucinated fields.
- Route low-confidence cases to human review; corrections retrain the models and raise the automation rate.