Observability and Evaluation for AI in Production
AI fails silently. GPU metrics, latency SLOs, OpenTelemetry tracing, online evals and cost tracking for production AI systems.
Key takeaways
- AI systems fail silently: an HTTP 200 can carry a wrong, unsafe or expensive answer.
- Observe three layers — infrastructure (GPUs), serving (latency, queues, tokens) and application (traces, quality, cost).
- Distributed tracing with OpenTelemetry captures every model call, tool call and retrieval step in an agent run.
- Combine offline evals (before release) with online evals (sampled production traffic) and human review.
- Alert on symptoms users feel: latency SLOs, error spikes, quality drops and cost per task.
Traditional software tends to fail loudly — exceptions, crashes, 500 errors. AI systems fail quietly. A model can respond quickly and successfully while giving a wrong answer, ignoring an instruction, leaking sensitive data or burning ten times the expected tokens in an agent loop. Observability for AI therefore has to go beyond uptime dashboards. This article covers what to measure, how to trace it and how to evaluate quality continuously.
Three layers of observability
Infrastructure: GPU metrics
NVIDIA's DCGM exporter publishes GPU telemetry to Prometheus: utilization, memory usage, power draw, temperature, clock speeds and error counters. Hardware errors (such as ECC or Xid errors) are an early warning that a GPU or node should be drained — important when a single failure can stall a multi-GPU job.
Serving: the latency metrics that matter
| Metric | What it tells you |
|---|---|
| TTFT (time to first token) | Perceived responsiveness; driven by queueing and prefill |
| ITL / TPOT (inter-token latency / time per output token) | Streaming smoothness; driven by decode speed and batch size |
| End-to-end latency | Total time, dominated by output length |
| Queue depth / waiting requests | Saturation — the best leading indicator for autoscaling |
| Tokens in / out per second | Load and capacity |
| KV-cache usage and prefix-cache hit rate | Memory pressure and caching effectiveness |
Always track percentiles (p50, p95, p99), not averages, and segment by model, route and input-length bucket. Example PromQL queries against a vLLM server:
# p95 time to first token over 5 minutes
histogram_quantile(0.95, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m])))
# requests waiting in the queue (saturation)
sum(vllm:num_requests_waiting)
# output tokens per second across the fleet
sum(rate(vllm:generation_tokens_total[1m]))
Application: tracing agent runs
A single user request to an AI feature may involve retrieval, several model calls, tool executions and retries. Distributed tracing captures this as a tree of spans so you can see exactly what happened and where time and tokens went.
The industry is converging on OpenTelemetry, which has semantic conventions for generative AI (gen_ai.* attributes such as model name, token usage and operation type). Many SDKs and frameworks emit these spans, and tools like Langfuse, Arize Phoenix, LangSmith and Datadog can ingest and visualize them.
from opentelemetry import trace
tracer = trace.get_tracer("support-assistant")
def answer(question: str) -> str:
with tracer.start_as_current_span("answer_question") as span:
span.set_attribute("app.prompt_version", "support-v14")
docs = retrieve(question) # traced as a child span
span.set_attribute("app.retrieved_docs", len(docs))
result = call_model(question, docs) # model client emits gen_ai.* spans
span.set_attribute("gen_ai.usage.input_tokens", result.input_tokens)
span.set_attribute("gen_ai.usage.output_tokens", result.output_tokens)
return result.text
Record the configuration version (model ID, prompt version, retrieval index) on every trace. When quality drops, the first question is always "what changed?"
Evaluating quality
Offline evals — before release
Run a curated dataset through every candidate configuration in CI. Use deterministic graders wherever possible (valid JSON, correct answer, tests pass) and an LLM judge with a clear rubric for open-ended outputs. Calibrate the judge against human labels so you know how much to trust it. See MLOps in the LLM Era for release pipelines.
Online evals — in production
- Sampled LLM-as-judge — score a percentage of live traffic for correctness, groundedness, tone and policy compliance.
- Implicit signals — retries, rephrased questions, abandoned sessions, escalations to humans, edits to generated text.
- Explicit feedback — thumbs up/down and comments, routed into a review queue.
- Task outcomes — for agents: did the ticket get resolved, did the tests pass, was the PR merged?
Human review
Automated scores drift and miss nuance. Schedule regular human review of sampled conversations, especially failures. The best eval datasets are built from this review: every production failure becomes a regression test.
Cost observability
Token spend can grow unnoticed — a prompt change that doubles context length, an agent that loops, a cache that stops hitting. Track:
- Cost per request, per user, per feature and per completed task.
- Input vs output vs cached tokens, and cache hit rates.
- Agent steps per task and the distribution's long tail.
- For self-hosted models: GPU-hours and cost per million tokens served.
Alerting that works
Alert on symptoms users feel, with thresholds tied to SLOs:
- p95 TTFT or end-to-end latency above SLO for 10 minutes.
- Error or timeout rate spikes, including rate-limit errors from upstream APIs.
- Online quality score dropping below baseline after a deploy.
- Cost per task or token usage exceeding budget.
- GPU hardware errors or nodes failing health checks.
Combined with safe rollouts and versioned configuration, these signals turn AI from a black box into a system you can operate with confidence.