Home / Articles / MLOps
MLOps Engineering

Observability and Evaluation for AI in Production

AI fails silently. GPU metrics, latency SLOs, OpenTelemetry tracing, online evals and cost tracking for production AI systems.

By SiliconMind EditorialNov 202612 min read
Key takeaways
  • AI systems fail silently: an HTTP 200 can carry a wrong, unsafe or expensive answer.
  • Observe three layers — infrastructure (GPUs), serving (latency, queues, tokens) and application (traces, quality, cost).
  • Distributed tracing with OpenTelemetry captures every model call, tool call and retrieval step in an agent run.
  • Combine offline evals (before release) with online evals (sampled production traffic) and human review.
  • Alert on symptoms users feel: latency SLOs, error spikes, quality drops and cost per task.

Traditional software tends to fail loudly — exceptions, crashes, 500 errors. AI systems fail quietly. A model can respond quickly and successfully while giving a wrong answer, ignoring an instruction, leaking sensitive data or burning ten times the expected tokens in an agent loop. Observability for AI therefore has to go beyond uptime dashboards. This article covers what to measure, how to trace it and how to evaluate quality continuously.

Three layers of observability

Applicationtraces · prompts & outputs · tool calls · quality scores · cost per task · user feedback ServingTTFT · inter-token latency · queue depth · tokens in/out · KV-cache usage · errors InfrastructureGPU utilization · HBM usage · power & temperature · ECC/Xid errors · node health
Fig. 1 — Each layer answers a different question: is the hardware healthy, is serving fast, and are the answers good?

Infrastructure: GPU metrics

NVIDIA's DCGM exporter publishes GPU telemetry to Prometheus: utilization, memory usage, power draw, temperature, clock speeds and error counters. Hardware errors (such as ECC or Xid errors) are an early warning that a GPU or node should be drained — important when a single failure can stall a multi-GPU job.

Utilization can misleadA "100% GPU utilization" metric only means a kernel was running, not that the GPU was doing useful work efficiently. For inference, throughput in tokens per second and cost per token are better efficiency measures.

Serving: the latency metrics that matter

MetricWhat it tells you
TTFT (time to first token)Perceived responsiveness; driven by queueing and prefill
ITL / TPOT (inter-token latency / time per output token)Streaming smoothness; driven by decode speed and batch size
End-to-end latencyTotal time, dominated by output length
Queue depth / waiting requestsSaturation — the best leading indicator for autoscaling
Tokens in / out per secondLoad and capacity
KV-cache usage and prefix-cache hit rateMemory pressure and caching effectiveness

Always track percentiles (p50, p95, p99), not averages, and segment by model, route and input-length bucket. Example PromQL queries against a vLLM server:

promql
# p95 time to first token over 5 minutes histogram_quantile(0.95, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m]))) # requests waiting in the queue (saturation) sum(vllm:num_requests_waiting) # output tokens per second across the fleet sum(rate(vllm:generation_tokens_total[1m]))

Application: tracing agent runs

A single user request to an AI feature may involve retrieval, several model calls, tool executions and retries. Distributed tracing captures this as a tree of spans so you can see exactly what happened and where time and tokens went.

The industry is converging on OpenTelemetry, which has semantic conventions for generative AI (gen_ai.* attributes such as model name, token usage and operation type). Many SDKs and frameworks emit these spans, and tools like Langfuse, Arize Phoenix, LangSmith and Datadog can ingest and visualize them.

python
from opentelemetry import trace tracer = trace.get_tracer("support-assistant") def answer(question: str) -> str: with tracer.start_as_current_span("answer_question") as span: span.set_attribute("app.prompt_version", "support-v14") docs = retrieve(question) # traced as a child span span.set_attribute("app.retrieved_docs", len(docs)) result = call_model(question, docs) # model client emits gen_ai.* spans span.set_attribute("gen_ai.usage.input_tokens", result.input_tokens) span.set_attribute("gen_ai.usage.output_tokens", result.output_tokens) return result.text

Record the configuration version (model ID, prompt version, retrieval index) on every trace. When quality drops, the first question is always "what changed?"

Privacy firstPrompts and outputs often contain personal or confidential data. Decide deliberately what to log, redact sensitive fields, restrict access and set retention limits — before turning on full-content tracing.

Evaluating quality

Offline evals — before release

Run a curated dataset through every candidate configuration in CI. Use deterministic graders wherever possible (valid JSON, correct answer, tests pass) and an LLM judge with a clear rubric for open-ended outputs. Calibrate the judge against human labels so you know how much to trust it. See MLOps in the LLM Era for release pipelines.

Online evals — in production

  • Sampled LLM-as-judge — score a percentage of live traffic for correctness, groundedness, tone and policy compliance.
  • Implicit signals — retries, rephrased questions, abandoned sessions, escalations to humans, edits to generated text.
  • Explicit feedback — thumbs up/down and comments, routed into a review queue.
  • Task outcomes — for agents: did the ticket get resolved, did the tests pass, was the PR merged?

Human review

Automated scores drift and miss nuance. Schedule regular human review of sampled conversations, especially failures. The best eval datasets are built from this review: every production failure becomes a regression test.

Cost observability

Token spend can grow unnoticed — a prompt change that doubles context length, an agent that loops, a cache that stops hitting. Track:

  • Cost per request, per user, per feature and per completed task.
  • Input vs output vs cached tokens, and cache hit rates.
  • Agent steps per task and the distribution's long tail.
  • For self-hosted models: GPU-hours and cost per million tokens served.

Alerting that works

Alert on symptoms users feel, with thresholds tied to SLOs:

  • p95 TTFT or end-to-end latency above SLO for 10 minutes.
  • Error or timeout rate spikes, including rate-limit errors from upstream APIs.
  • Online quality score dropping below baseline after a deploy.
  • Cost per task or token usage exceeding budget.
  • GPU hardware errors or nodes failing health checks.

Combined with safe rollouts and versioned configuration, these signals turn AI from a black box into a system you can operate with confidence.