Production requirement · 05
Latency & cost
Budgets agreed before we build. Routing, caching and smaller models are used wherever they meet the quality bar — and measured when they do not.
The largest model at the lowest temperature is the easy way to win a demo and the quick way to lose a business case. Latency and cost are constraints we agree up front and design against, the same way we would for any other production service.
What it means to us
- Budgets are inputs, not outcomes. Before the first line of code we agree a latency target at a percentile (usually p95), a monthly cost ceiling and the volume they assume. Every design choice is checked against them.
- Route to the smallest thing that passes. Many requests do not need the strongest model. A cheaper model, a cached answer or a rule handles them when — and only when — the evaluation set shows it meets the bar. The routing decision is a traced, measured component.
- Cache what is stable, stream what is slow. Retrieval results, embeddings and repeated prompts are cached with explicit invalidation. Where a user is waiting, we stream partial results rather than optimise a single blocking call.
- Load-test before launch. We measure the system at the agreed volume, including the review queue and any downstream tools, and publish the numbers alongside the quality scorecard.
- Cost per outcome, not per token. We report the cost of a decided document or a resolved ticket so trade-offs between quality and spend are made in the business’s units.
What we deliver
Written budgets; a routing and caching design with its own evaluation; load-test results; and cost dashboards that break spend down per feature and per route.
What it is not
Budgets are not guarantees against a provider changing its prices or a workload changing shape. They are the instrument that shows the change early and the design that gives you options when it happens.