AI document extraction and OCR: from paper and PDF to structured data

Back offices, shared service centres, accountancy firms, insurers and legal teams process thousands of invoices, policies, contracts, KvK extracts (Dutch Chamber of Commerce), valuations and email attachments every day. Traditional OCR recognises characters but leaves the interpretation to people. With modern AI document extraction (Vision LLMs, layout-aware models and structured-output schemas), software reads documents in any format and delivers validated JSON straight to your accounting, policy or case management system.

Invoice extraction (PEPPOL UBL) KvK extracts Policy and claims processing Contract clause extraction Passport/ID and KYC CVs and emails
Discuss your document flow View use cases
AI { "invoice_no": "F-2419" "vendor": "Acme BV" "vat_id": "NL8001…" "date": "2026-04-30" "total": 1842.50 "currency": "EUR" "line_items": […] "confidence": 0.97 }

What AI document extraction and modern OCR do

Document processing has changed fundamentally over the past five years. Where Tesseract and ABBYY used to read characters and you had to write regular expressions and templates yourself, Vision LLMs and layout-aware models now extract meaning from a document, including tables, handwritten notes, stamps and scanned copies of varying quality.

An invoice can arrive as a structured PEPPOL UBL file, as a PDF generated from an ERP system, as a scanned paper invoice or as a photo taken on a phone. For every format, your system needs to correctly identify the supplier, VAT number, invoice lines, final totals and payment reference, and ideally also detect whether the booking lines make sense. Classic OCR delivers text, not meaning. AI document extraction delivers a validated JSON object that matches the schema your accounting package or case management system expects.

The same applies to KvK extracts, land registry deeds and documents, policy schedules, claim files, purchase agreements, mortgage quotes, passports and ID documents, CVs, receipts and even the content of incoming emails. Each type of document has its own schema. Appfront designs that schema with you, trains or fine-tunes the model and builds the pipeline that receives, classifies, extracts, validates and forwards documents to your existing systems, with a human in the loop for cases where the confidence score falls below your threshold.

The difference from traditional OCR comes down to three things: layout awareness (models understand columns, tables and visual grouping), context understanding (the word "total" is weighted differently in the bottom right than in the header) and direct structure (the output is a typed JSON schema, not a block of plain text you still have to parse yourself). That difference determines whether processing takes 30 seconds and then flows through automatically in 90% of cases, or whether every invoice still passes through human hands.

Document types we process

We build extraction pipelines for almost every document found in a Dutch back office. Below are the six most common categories — for each of them we have a reference schema that we refine together with you.

📄

Invoices and receipts (AP automation)

PEPPOL UBL via government or supplier portals, free-form PDF invoices, scanned paper invoices, till receipts and restaurant receipts. We extract the supplier, KvK and VAT numbers, IBAN, invoice number, invoice date, due date, line items with VAT rate and final totals. The output connects to Exact Online, AFAS, Twinfield, Yuki, Visma.net or your own ERP. See also our invoice recognition page.

🏛️

KvK extracts and land registry documents

Extracts from the Trade Register, land registry notices, property ownership information, mortgage deeds and deeds of transfer. Alongside direct extraction via the KvK API and the Land Registry integration, we also process scanned historical files where no API source is available.

📋

Policies and claims files

Policy schedules, policy terms and conditions, claim forms (police reports, expert reports), medical reports and correspondence from insurance intermediaries. The model recognises coverages, excess, premiums, policy numbers and claim items. Suitable for health insurers, general and life insurers, reinsurers and managing agents.

⚖️

Contracts and legal files

Purchase agreements, employment contracts, NDAs, SLAs, tenancy agreements and general terms and conditions. Clause extraction identifies notice periods, penalty clauses, governing-law clauses, price indexation and termination conditions. The output goes to your contract management system or CLM tooling.

🆔

Passports, IDs and KYC documents

Dutch and foreign passports, ID cards, driving licences, MRZ extraction, NFC reading where available, plus supplementary KYC evidence such as utility bills and municipal personal records database (GBA) extracts. Part of broader KYC and AML compliance software for banks, insurers and notaries.

📧

CVs, emails and attachments

CVs in any format (PDF, Word, scanned), incoming emails with multiple attachments, freight documents, CMR forms and specification PDFs for construction and infrastructure projects. The pipeline first classifies the document type, picks the right schema and then extracts. For the construction sector, see also our CMR/freight document page.

OCR, ICR, IDP and vision LLMs: what works for what

The field has jargon that tends to get mixed up. A short orientation helps you decide which type of solution suits your document flow — and which combination you will usually need in production.

Traditional OCR (Tesseract, ABBYY FineReader)

Classic optical character recognition reads pixels and returns text. It works well for clean scans of typed documents. The downsides: no understanding of layout, no interpretation of tables, and sensitivity to skewed scans and handwriting. Usable as a pre-processor for downstream NLP, but it requires regex engineering or templates for structure.

ICR (Intelligent Character Recognition)

A variant of OCR that also tries to recognise handwritten characters. Old-fashioned ICR engines are limited; modern handwriting recognition relies on transformer models. Relevant for claim forms, medical notes and historical archives.

IDP (Intelligent Document Processing)

Umbrella term for end-to-end platforms: Hyperscience, Rossum, AWS Textract, Azure Document Intelligence and Google Document AI. They combine OCR, layout analysis, classification and extraction. A good fit if you want a fast, out-of-the-box route and process standard document types; less suitable for unique schemas or strict EU data residency requirements.

Vision LLMs and layout-aware models

Donut, LayoutLMv3, Pix2Struct and the current vision models from Anthropic Claude and OpenAI GPT-4o read documents holistically: image, text and layout at the same time. They deliver structured JSON output directly via structured-output schemas. The advantage: no regex, no template engineering and fast iteration. The point to watch: cost per page and data residency call for deliberate architecture choices.

In practice, we often combine several layers: a lightweight OCR or layout analyser (for example Azure Document Intelligence Read or an open-source stack) for preprocessing, a classifier that determines the document type, and then a specialised extraction model. Sometimes that is a fine-tuned LayoutLMv3 for high-volume standard documents, sometimes a vision LLM for the long tail and exceptions. In our experience, this combination gives the best balance between cost, latency and accuracy.

Which approach suits you depends on volume, data sensitivity, the straight-through processing rate you want and your existing infrastructure. In an intake conversation we map out these variables and make a proposal with a proof of concept. See also our page on the AI discovery workshop.

Applications by sector

Document extraction has its own character in every sector. Below are four sectors where we work regularly; the patterns apply more broadly.

Accountancy and bookkeeping

For accountancy firms and bookkeeping departments, we automate accounts payable: incoming invoices are read, matched to purchase orders, posted to the correct general ledger account (based on supplier and previous postings) and passed on to the approver. Can be combined with our AI for accountancy page and accountants' portal.

Insurance and insurance intermediaries

Policy entry and claims handling are document-heavy. The model reads policy schedules from mandated insurers, classifies claims based on damage reports and routes them to the right handler. See also AI for insurance and insurance app development.

Legal and notarial

Legal teams use clause extraction for due diligence (data rooms with hundreds of contracts), contract review and risk flagging. For notary offices, we extract deeds, land registry documents and KvK (Chamber of Commerce) extracts. The output flows into your DMS or CLM.

Shared service centres and back office

At large organisations with a central back office (energy, telecom, housing associations, care providers), we process email correspondence, customer letters and forms. The pipeline first classifies (quote, complaint, contract change), then extracts and routes to the right department. Often linked to workflow engines through our document workflow automation.

How we deliver a document extraction pipeline

From initial analysis to production in four phases. We work iteratively and validate every component with real documents from your own archive.

Document audit

You provide a representative set of a few hundred documents. We analyse formats, quality and layout variation, and determine which document types account for the most volume. The result: a priority list and an initial schema proposal.

Schema and golden dataset

For each document type, we define a JSON schema (required fields, types, validations). You label or validate a golden dataset of 100-500 documents. This becomes our ground truth for both training and testing.

Model selection and pipeline

We choose between a Vision LLM, fine-tuned LayoutLMv3/Donut or a hybrid. Around that we build the pipeline: intake (email, API, upload), preprocessing, classification, extraction, schema validation and human-in-the-loop review for low-confidence cases.

Production and monitoring

Deployment in your cloud or on-premises, integration with downstream systems, monitoring of accuracy and throughput, and a feedback loop in which corrected documents continuously improve the model. Includes alerting when accuracy drops below a threshold.

Not sure about a large project yet?

Test your idea first: a working prototype in 1 day

With OneDayBuild, we make your idea tangible in a single day for €1,150, so you know whether further development is worth the investment. Decide to go ahead with the full build? Then we deduct the cost in full.

View OneDayBuild →

Technology and tooling

We are vendor-agnostic and choose based on your requirements for data residency, cost and accuracy. Below is the stack that comes up regularly — your situation may call for a different combination.

Tesseract ABBYY FineReader AWS Textract Azure Document Intelligence Google Document AI Hyperscience Rossum LayoutLMv3 Donut Pix2Struct Claude Vision GPT-4o Hugging Face Transformers PyTorch LangChain LlamaIndex Pydantic FastAPI Docker PostgreSQL PEPPOL UBL NEN 2082 (e-archive)

Alongside model engineering, we pay close attention to structured-output schemas (Pydantic, JSON Schema, Anthropic tool use or OpenAI function calling), retry and validation logic, and archive compliance under NEN 2082 for organisations that must retain documents in line with Dutch e-archiving legislation. Where relevant, we also address eIDAS aspects (signature validation on contracts, integrity of deeds).

If you use Hyarchis for documents in the mortgage process, see our page on a Hyarchis integration.

If you would rather integrate an existing document recognition platform than build your own, see our page on a Klippa integration.

Architecture, accuracy and human-in-the-loop

A document extraction system that extracts with 95% accuracy gets something wrong in the remaining 5%. The question is how you spot those errors, validate them and feed them back into the model. Our architecture is designed around exactly that.

Confidence scores and thresholds

We provide a confidence score for each field. Fields below your threshold (for example 0.90 for IBAN, 0.98 for total amount) go to a review queue. Fields above the threshold flow straight through. You decide the balance between straight-through processing and quality assurance.

Human-in-the-loop interface

A lightweight review interface shows the document with the extracted fields and their detected positions. Your staff correct in seconds rather than minutes. Every correction is recorded as an improvement signal for model retraining or prompt tuning.

Schema validation and business rules

Beyond confidence, we validate the output: do the line items add up to the total, is the VAT rate consistent, is the IBAN check digit correct, does the KvK number exist? Business rules catch plausible but incorrect extractions that the model cannot spot on its own.

Continuous evaluation

A dashboard tracks field-level accuracy, turnaround time, escalations and cost per document — over time and per document type. A regression triggers an alert. We use golden datasets for regression tests with every model update.

For sectors with strict data residency requirements (healthcare, government, financial institutions), we build the extraction pipeline on your own infrastructure or a European cloud — no documents leave your control. For less sensitive flows, a Vision LLM as a service can be used within Europe, provided a DPA is in place and zero-retention is enabled. We will discuss which route fits during the architecture phase.

Why Appfront for document extraction

Understanding document flows

We have built pipelines for accountancy firms, insurers, legal teams, housing associations and logistics providers. That domain knowledge translates into pragmatic schemas and realistic expectations around accuracy.

No vendor lock-in

We choose models and cloud providers case by case. No forced Hyperscience or Rossum licence if an open-source LayoutLMv3 on your own infrastructure is a better fit. Your schema and your data remain yours.

From POC to production

Many document AI projects stall between prototype and actual integration with an ERP, DMS or policy system. We support the entire journey: extraction, integration, monitoring and ongoing maintenance.

Frequently asked questions about AI document extraction

What is the difference between classic OCR and AI document extraction?
Classic OCR (Tesseract, ABBYY) converts pixels to text, without any understanding of layout or meaning. AI document extraction combines layout analysis, classification and interpretation, and directly returns a structured JSON object that matches your target schema. In practice, modern pipelines often use a combination: OCR as preprocessing, then a Vision LLM or layout-aware model for the extraction.
What level of accuracy is achievable?
That depends on the document type, the quality of the scans and the complexity of the schema. For standard invoices, we generally achieve high accuracy on key fields such as supplier, total amount and date. For complex contracts or poorly scanned historical files, it is lower. We measure and report field-level accuracy on a golden dataset; that is the honest yardstick.
Can this run on-premises for data residency reasons?
Yes. For sectors where data may not leave the premises (healthcare, government, some financial institutions), we build the pipeline on your own infrastructure or a European cloud. Open-source models such as LayoutLMv3, Donut or a local Llama model can run fully self-hosted.
How do you handle PEPPOL UBL and structured formats?
PEPPOL UBL and e-invoice XML are already structured; they need no extraction, only a mapping to your internal schema. Our pipeline classifies the format first. With structured input, it skips the extraction phase and validates straight away. With free-form PDFs or scans, the document goes through the extraction models. The result: the same schema, regardless of the input format.
Does it also work for handwritten claim forms?
Handwritten text is harder than typed text, but modern Vision LLMs and specialised handwriting recognition models achieve reasonable results on legible forms. For critically important fields (such as the claim amount), we almost always recommend human-in-the-loop review.
How does it connect to our accounting or policy software?
We build integrations via REST APIs, webhooks or existing integration platforms. For accounting packages such as Exact Online, AFAS, Twinfield, Yuki and Visma.net we have standard mappers; for policy systems we work with Sequel, IDIT and in-house policy administrations. The extraction delivers the schema, the integration carries it through.
Does this comply with GDPR and NEN 2082?
GDPR: documents may contain personal data; we set up the pipeline according to privacy by design, with a DPA, zero retention at external model providers and clear retention periods. NEN 2082 (e-archiving): we implement metadata, integrity safeguards and durable storage where you are subject to archiving obligations. The legal context determines the exact implementation.
How long does an implementation typically take?
A proof of concept for a single document type is usually ready within a few weeks. A production pipeline with multiple document types, integrations and monitoring takes longer, depending heavily on the complexity of your system landscape. We work in iterative sprints with interim deliveries, so you can see early on whether the approach works.

Want to automate your document flow?

Discuss your document flow with Appfront. Together we analyse which document types take the most time and where AI extraction delivers the biggest gains, with no obligation.

Schedule a call

Edit content