Measuring accuracy in document extraction

"How accurate is it?" is the first question every client asks about document extraction, and it is usually asked as if the answer were a single number. It should not be.

Why a single accuracy score misleads

A document-extraction system typically pulls dozens of fields from each document — invoice number, vendor name, line-item amounts, dates, tax codes. Averaging accuracy across all of them hides the fields that matter most. A system that is 98% accurate on "document type" but 80% accurate on "total amount" is not a 90%-accurate system in any useful sense — it is a system you cannot trust with the one field that drives downstream decisions.

The right approach is to report accuracy per field, on a labelled sample large enough to be statistically meaningful for that field's frequency. Rare fields need proportionally larger samples to produce a trustworthy number.

Precision, recall, and the cost of being wrong

For each field, we track two numbers, not one:

• Precision — of the values the system extracted, how many were correct.

• Recall — of the values that were actually present in the document, how many did the system find.

These trade off against each other, and the right balance depends on what happens next. For a field that flows straight into an automated payment, you want very high precision — a wrong number extracted with confidence is worse than a value flagged for review. For a field used only to route a document to the right queue, recall matters more; a missed classification just means a slower path, not a wrong outcome.

Confidence scores are a promise you have to keep

Most extraction models output a confidence score alongside each value. That score is only useful if it is calibrated — meaning a field extracted at "90% confidence" really is correct about 90% of the time, not 60% or 99%. We test this explicitly: bucket extracted values by their confidence score, then check the actual accuracy within each bucket. If the buckets don't line up with the confidence numbers, the threshold used to route items to human review is set on faith, not evidence.

What this looks like before go-live

Before any document-intelligence system goes live, we run it against a labelled sample of the client's own documents — not a public benchmark dataset, which tells you nothing about a specific vendor's invoice layout or a specific court's filing format. We report:

• Field-level precision and recall

• The confidence threshold chosen for auto-approval versus human review, and the accuracy at that threshold

• A breakdown by document variant, since a single "document type" (an invoice, a lease) often contains several visually distinct templates that perform differently

This is more work up front than quoting a single headline number. It is also the only way to know, before a client depends on it, which decisions the system is actually ready to make on its own — and which ones still need a person to look first.