MLOps in the LLM Era: From Notebook to Production
Versioning, evaluation gates, progressive delivery and monitoring — how teams ship and operate AI models reliably.
Key takeaways
- MLOps is the discipline of shipping models reliably: versioned data and models, automated pipelines, evaluation gates and monitoring.
- LLMs shift the work from training models to managing prompts, retrieval, fine-tunes, evaluations and inference cost — often called LLMOps.
- Treat prompts, model versions and evaluation datasets as code: version them, review them and test them in CI.
- Evaluation gates and progressive delivery (shadow, canary, A/B) are what make model releases safe.
- Production monitoring must cover quality, latency, cost and safety — not just uptime.
Most machine learning projects that fail don't fail because the model was bad. They fail between the notebook and production: the data pipeline breaks, nobody can reproduce last month's model, a new version silently performs worse, or inference costs spiral. MLOps is the set of practices that closes that gap. The rise of large language models has changed what MLOps teams spend their time on — but made the discipline more important, not less.
The ML lifecycle
Each stage needs automation and traceability:
- Data — versioned datasets (DVC, lakeFS, Delta/Iceberg tables), validation checks and lineage.
- Training and tuning — reproducible pipelines (Kubeflow Pipelines, Argo Workflows, Ray, SageMaker or Vertex AI Pipelines) with experiment tracking (MLflow, Weights & Biases).
- Evaluation — automated test suites that decide whether a candidate is good enough.
- Model registry — the source of truth for which model version is in which environment, with metadata, approvals and rollback.
- Deployment — serving infrastructure, often on Kubernetes (see Serving LLMs on Kubernetes).
- Monitoring — latency, errors, cost, data drift and output quality.
What changes with LLMs
Classic MLOps assumed you train your own model on your own data. With foundation models, most teams start from a pre-trained model — via an API or open weights — and the artifacts that matter shift:
| Classic ML | LLM applications (LLMOps) |
|---|---|
| Feature pipelines | Retrieval pipelines, chunking and embeddings |
| Training from scratch | Prompting, fine-tuning (LoRA), distillation |
| Accuracy, AUC on a test set | Task-specific evals, LLM-as-judge, human review |
| Model binary is the artifact | Prompt + model ID + tools + retrieval config is the artifact |
| Cheap inference | Inference cost and latency dominate |
| Data drift | Drift plus model-version changes, prompt injection, hallucination |
Version everything that changes behavior
An LLM feature's behavior is determined by several moving parts. If any of them changes without review, quality can regress silently:
- Model identity — pin exact model versions, never "latest." Upgrading a model is a release.
- Prompts and system instructions — store them in git, review changes like code, and tag them with versions.
- Tool definitions — names, descriptions and schemas affect what the model does.
- Retrieval configuration — embedding model, chunk size, index version, top-k.
- Generation settings — effort or reasoning level, max output length, structured-output schemas.
- Evaluation datasets — the test set is part of the definition of "good."
Evaluation gates in CI/CD
The heart of LLMOps is the eval suite: a curated set of inputs with graders, run automatically on every change. Graders can be exact checks (valid JSON, correct SQL result, passing unit tests), heuristic checks, or a strong model acting as a judge against a rubric. A release pipeline might look like this:
# .github/workflows/llm-release.yml
name: llm-release
on:
pull_request:
paths: ["prompts/**", "config/model.yaml", "retrieval/**"]
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install -r eval/requirements.txt
# run the candidate config against the golden dataset and compare to production
- run: python eval/run_suite.py --candidate config/model.yaml --baseline prod --min-pass-rate 0.92
- run: python eval/cost_check.py --max-cost-per-task-usd 0.05
deploy-canary:
needs: evaluate
if: github.event.pull_request.merged == true
runs-on: ubuntu-latest
steps:
- run: ./deploy/canary.sh --traffic 5 # shift 5% of traffic to the new version
Good eval suites share a few traits: they're built from real production examples, they include known past failures as regression tests, they're small enough to run on every change, and they're re-run several times for agentic tasks where behavior varies.
Progressive delivery for models
Offline evals can't catch everything. Production rollouts reduce risk with techniques borrowed from software delivery:
- Shadow deployment — send a copy of live traffic to the new version without showing its responses to users; compare outputs offline.
- Canary — route a small percentage of traffic to the new version and watch quality, latency and error metrics before increasing.
- A/B testing — measure business outcomes (task completion, user ratings, conversion) between versions.
- Feature flags and instant rollback — switch model versions without redeploying the application.
On Kubernetes, tools like Argo Rollouts and Flagger automate canary analysis, and an AI gateway layer can split traffic between model versions or providers.
Fine-tuning pipelines
When prompting and retrieval aren't enough — for a specialized output format, domain vocabulary or lower latency with a smaller model — teams fine-tune. Parameter-efficient methods like LoRA train small adapter matrices instead of all weights, making it practical to maintain many task-specific variants. Serving engines like vLLM can load multiple LoRA adapters on one base model, so each variant doesn't need its own GPUs.
A fine-tuning pipeline needs the same rigor as any training pipeline: dataset versioning, held-out evaluation, comparison against the base model plus a good prompt, and registry promotion only when it wins.
Monitoring in production
Uptime is necessary but nowhere near sufficient. A model can return HTTP 200 for every request while giving wrong answers. Monitor four dimensions:
- Performance — time to first token, tokens per second, end-to-end latency, error and timeout rates.
- Cost — tokens per request, cost per user and per completed task, cache hit rates.
- Quality — sampled LLM-as-judge scores, user feedback, task success, escalation rates.
- Safety — refusals, policy violations, prompt-injection attempts, PII leakage.
We go deeper into tooling in Observability and Evaluation for AI in Production.
The MLOps platform landscape
| Need | Open source | Managed |
|---|---|---|
| Experiment tracking & registry | MLflow | Weights & Biases, SageMaker Model Registry, Vertex AI Model Registry |
| Pipelines | Kubeflow Pipelines, Argo Workflows, Airflow, Ray | SageMaker Pipelines, Vertex AI Pipelines, Azure ML |
| Serving | vLLM, SGLang, KServe, Triton, Ray Serve | Bedrock, SageMaker endpoints, Vertex AI endpoints |
| LLM tracing & evals | Langfuse, Arize Phoenix, OpenTelemetry | LangSmith, Braintrust, cloud-native options |
There's no single correct stack. Small teams often do well with a managed model API, prompts in git, a lightweight eval harness in CI and an observability tool. Teams self-hosting open models on GPUs need the full platform — which is where Kubernetes comes in.