Production requirement ยท 02
Human oversight
Explicit confidence thresholds and review queues. The system recommends; a named person stays accountable for every decision.
Most AI systems in a business make or shape decisions that someone is already accountable for. Oversight is the design work that keeps that accountability intact when part of the work is now done by a probabilistic component.
What it means to us
- Confidence is a first-class output. Every prediction carries a calibrated confidence, and we measure whether that confidence means what it says. A system that is 90% sure should be right about nine times in ten on the evaluation set.
- Thresholds are explicit and owned. The line between “act automatically” and “queue for a person” is a documented number with a named owner. It moves as the evaluation numbers move, and the change is logged.
- Review is a designed workflow, not a fallback. The queue a reviewer sees shows the document, the system’s suggestion, the evidence behind it and a one-click way to agree or override. Reviewer time is the scarcest resource in the loop; we design for it.
- Overrides feed back. Every human correction is captured with context and becomes a candidate for the evaluation set. The queue shrinks because the system earns trust, not because the threshold was quietly lowered.
- Nothing acts alone where a person must sign. Where a regulation, contract or policy requires human judgment, the system pre-fills and recommends. It never presses the button.
What we deliver
A routing policy document; the review interface or an integration into the tooling reviewers already use; an override log with reporting; and a monthly threshold review as part of operations.
What it is not
Oversight is not a rubber stamp or a way to shift blame onto a reviewer who cannot see the evidence. If the queue is too large for the people available, that is a capacity problem we surface, not one we hide.