Production requirement · 04
Observability
A trace for every model call and tool use, cost and latency per request, and drift dashboards wired into the tooling your team already watches.
A deterministic service fails loudly. A probabilistic one degrades quietly: answers get a little worse, a tool starts timing out, a prompt change shifts the distribution of outputs. Observability is how we make that degradation visible in the same place your team already looks for trouble.
What it means to us
- One trace per request, end to end. Every model call, retrieval, tool invocation and human hand-off is a span with its inputs, outputs, latency and cost. A reviewer or engineer can replay any decision after the fact.
- In your tooling, not ours. Traces and metrics export to the observability stack the team already uses — whether that is OpenTelemetry-compatible tooling, a cloud provider’s suite or a self-hosted setup. We do not introduce a second pane of glass that nobody watches.
- Drift is a metric. Confidence distributions, override rates, retrieval hit rates and evaluation-sample accuracy are plotted over time with alerts on the trend, not just the threshold.
- Cost is a first-class signal. Tokens, calls and compute roll up per feature and per customer segment so a runaway prompt shows up as a spend alert before the invoice does.
- Replay is built in. Traces are structured so a production request can be re-run offline against a new prompt or model version — which is how the evaluation set grows.
What we deliver
Instrumentation across the system; dashboards for quality, latency, cost and drift; alert rules with runbook links; and replay tooling that turns production traces into evaluation candidates.
What it is not
Observability does not make a system correct. It makes incorrectness visible fast enough to act on, which is the most a probabilistic system can promise.