Document decisioning with confidence routing

This write-up describes a representative engagement. Identifying details have been changed and the customer is not named; the architecture, constraints and evaluation approach are as delivered.

The problem

A finance operations team processed several hundred contracts and supplier invoices a week. Each document needed a handful of fields extracted, a category assigned and a decision made: approve, hold or escalate. A data scientist had built a promising notebook prototype using a hosted LLM. On a hand-picked sample it got roughly four in five documents right.

Nobody could say which four in five. There was no labelled set, no way to reproduce a result, and no plan for what happened to the one in five. The team was, reasonably, unwilling to let it near a real approval.

Constraints

• Data boundary. Documents contain pricing and counterparty information that must not leave the customer’s cloud account. Hosted model APIs were acceptable only through a private endpoint with no retention.
• Latency and cost. The workflow is interactive: a reviewer waits on the result. We set a budget of under one second per page and a fixed monthly cost ceiling.
• Accountability. A named human must remain responsible for every approval. The system could recommend and pre-fill, never decide alone.

What we built

1. An evaluation set first

Before touching the pipeline, we spent two weeks with the operations team building a labelled set: around 1,200 documents with ground-truth fields, categories and decisions, stratified by supplier type and document quality. It lives in the same repository as the code, is versioned, and every pull request runs against it.

2. Extraction and classification pipeline

• Layout-aware OCR feeding a retrieval step over the customer’s own supplier and contract records.
• Structured extraction with a schema-constrained model call, validated against business rules (totals reconcile, dates are in range, supplier exists).
• A small fine-tuned classifier for the category decision, which beat the general-purpose model on the evaluation set at a fraction of the latency.

3. Confidence routing

Each result carries calibrated confidence per field and for the overall decision. Above a threshold agreed with the team, the recommendation is pre-filled and the reviewer confirms with one action. Below it, the document is queued with the specific fields flagged for attention. Thresholds are configuration, not code, and are revisited monthly against the evaluation results.

4. Observability and operations

Every model call and rule evaluation emits a trace. Cost, latency and per-field accuracy on sampled production traffic feed dashboards in the tooling the customer’s platform team already watched. Reviewer overrides are captured and periodically folded back into the evaluation set.

Outcome

• Accuracy is now a measured number on a versioned set rather than an impression from a demo, and it is reported per category and per supplier type.
• The majority of documents are confirmed with a single reviewer action; the remainder arrive with the uncertain fields already highlighted.
• The review queue has shrunk as thresholds were tightened against evidence, and the customer’s own engineers operate the system using the runbooks we left behind.

Limitations

The system depends on the quality of the evaluation set; new supplier types need labelled examples before thresholds can be trusted for them. OCR quality remains the dominant source of error on scanned documents, and we recommend addressing capture upstream rather than compensating downstream.

What we reused afterwards

The evaluation harness structure, the confidence-routing pattern and the tracing conventions have since been applied on two further engagements with different document types. That reuse, more than any single model, is what makes the second project cheaper than the first.