Evaluation

A model that “seems to work” in a notebook is a hypothesis, not a result. Evaluation is how we turn the hypothesis into a number that a reviewer, a product owner and an engineer can all point at — and how we notice when that number moves.

What it means to us

• A labelled set comes first. Before we tune anything we sit with the people who do the job today and build a set of real cases with agreed answers. Disagreements between experts are recorded, not averaged away; they tell us where the task itself is ambiguous.
• The set is versioned with the code. Evaluation data lives in the repository (or a pointer to it does), changes go through review, and every result names the version it was measured on. “Accuracy went up” without a set version is not a claim we make.
• Offline first, then online. Every change runs against the held-out set in CI before anyone sees it. Once the system is live we sample production traffic, label a slice each week and compare it to the offline number so drift shows up as a trend, not a surprise.
• Metrics match the decision. A document router is judged on the cost of a wrong route, not on token-level F1. We agree on the metric that reflects the business consequence and report that one, even when a friendlier number is available.
• Scorecards, not screenshots. Demos show the system on cases it handles well. A scorecard shows it on the whole set, including the cases it does not. We ship the latter.

What we deliver

A first evaluation set during the framing phase; a scoring harness that runs locally and in CI; a scorecard template your team can keep publishing after we leave; and a written note on what the set does not cover.

What it is not

Evaluation is not a guarantee of future accuracy, and it is not a substitute for human oversight on decisions that matter. It is the instrument that tells you when the oversight threshold needs to move.