Home / Articles / MLOps
MLOps Engineering

MLOps in the LLM Era: From Notebook to Production

Versioning, evaluation gates, progressive delivery and monitoring — how teams ship and operate AI models reliably.

By SiliconMind EditorialNov 202613 min read
Key takeaways
  • MLOps is the discipline of shipping models reliably: versioned data and models, automated pipelines, evaluation gates and monitoring.
  • LLMs shift the work from training models to managing prompts, retrieval, fine-tunes, evaluations and inference cost — often called LLMOps.
  • Treat prompts, model versions and evaluation datasets as code: version them, review them and test them in CI.
  • Evaluation gates and progressive delivery (shadow, canary, A/B) are what make model releases safe.
  • Production monitoring must cover quality, latency, cost and safety — not just uptime.

Most machine learning projects that fail don't fail because the model was bad. They fail between the notebook and production: the data pipeline breaks, nobody can reproduce last month's model, a new version silently performs worse, or inference costs spiral. MLOps is the set of practices that closes that gap. The rise of large language models has changed what MLOps teams spend their time on — but made the discipline more important, not less.

The ML lifecycle

Data Train / tune Evaluate Registry Deploy Monitor quality · cost · drift FEEDBACK: new data, failures, user ratings
Fig. 1 — MLOps is a loop, not a line. Production signals feed the next iteration.

Each stage needs automation and traceability:

  • Data — versioned datasets (DVC, lakeFS, Delta/Iceberg tables), validation checks and lineage.
  • Training and tuning — reproducible pipelines (Kubeflow Pipelines, Argo Workflows, Ray, SageMaker or Vertex AI Pipelines) with experiment tracking (MLflow, Weights & Biases).
  • Evaluation — automated test suites that decide whether a candidate is good enough.
  • Model registry — the source of truth for which model version is in which environment, with metadata, approvals and rollback.
  • Deployment — serving infrastructure, often on Kubernetes (see Serving LLMs on Kubernetes).
  • Monitoring — latency, errors, cost, data drift and output quality.

What changes with LLMs

Classic MLOps assumed you train your own model on your own data. With foundation models, most teams start from a pre-trained model — via an API or open weights — and the artifacts that matter shift:

Classic MLLLM applications (LLMOps)
Feature pipelinesRetrieval pipelines, chunking and embeddings
Training from scratchPrompting, fine-tuning (LoRA), distillation
Accuracy, AUC on a test setTask-specific evals, LLM-as-judge, human review
Model binary is the artifactPrompt + model ID + tools + retrieval config is the artifact
Cheap inferenceInference cost and latency dominate
Data driftDrift plus model-version changes, prompt injection, hallucination

Version everything that changes behavior

An LLM feature's behavior is determined by several moving parts. If any of them changes without review, quality can regress silently:

  • Model identity — pin exact model versions, never "latest." Upgrading a model is a release.
  • Prompts and system instructions — store them in git, review changes like code, and tag them with versions.
  • Tool definitions — names, descriptions and schemas affect what the model does.
  • Retrieval configuration — embedding model, chunk size, index version, top-k.
  • Generation settings — effort or reasoning level, max output length, structured-output schemas.
  • Evaluation datasets — the test set is part of the definition of "good."
A useful mental modelTreat the combination of model + prompt + tools + retrieval config as a single deployable unit with a version number. Log that version with every request so production issues can be traced to an exact configuration.

Evaluation gates in CI/CD

The heart of LLMOps is the eval suite: a curated set of inputs with graders, run automatically on every change. Graders can be exact checks (valid JSON, correct SQL result, passing unit tests), heuristic checks, or a strong model acting as a judge against a rubric. A release pipeline might look like this:

yaml
# .github/workflows/llm-release.yml name: llm-release on: pull_request: paths: ["prompts/**", "config/model.yaml", "retrieval/**"] jobs: evaluate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: pip install -r eval/requirements.txt # run the candidate config against the golden dataset and compare to production - run: python eval/run_suite.py --candidate config/model.yaml --baseline prod --min-pass-rate 0.92 - run: python eval/cost_check.py --max-cost-per-task-usd 0.05 deploy-canary: needs: evaluate if: github.event.pull_request.merged == true runs-on: ubuntu-latest steps: - run: ./deploy/canary.sh --traffic 5 # shift 5% of traffic to the new version

Good eval suites share a few traits: they're built from real production examples, they include known past failures as regression tests, they're small enough to run on every change, and they're re-run several times for agentic tasks where behavior varies.

Progressive delivery for models

Offline evals can't catch everything. Production rollouts reduce risk with techniques borrowed from software delivery:

  • Shadow deployment — send a copy of live traffic to the new version without showing its responses to users; compare outputs offline.
  • Canary — route a small percentage of traffic to the new version and watch quality, latency and error metrics before increasing.
  • A/B testing — measure business outcomes (task completion, user ratings, conversion) between versions.
  • Feature flags and instant rollback — switch model versions without redeploying the application.

On Kubernetes, tools like Argo Rollouts and Flagger automate canary analysis, and an AI gateway layer can split traffic between model versions or providers.

Fine-tuning pipelines

When prompting and retrieval aren't enough — for a specialized output format, domain vocabulary or lower latency with a smaller model — teams fine-tune. Parameter-efficient methods like LoRA train small adapter matrices instead of all weights, making it practical to maintain many task-specific variants. Serving engines like vLLM can load multiple LoRA adapters on one base model, so each variant doesn't need its own GPUs.

A fine-tuning pipeline needs the same rigor as any training pipeline: dataset versioning, held-out evaluation, comparison against the base model plus a good prompt, and registry promotion only when it wins.

Monitoring in production

Uptime is necessary but nowhere near sufficient. A model can return HTTP 200 for every request while giving wrong answers. Monitor four dimensions:

  1. Performance — time to first token, tokens per second, end-to-end latency, error and timeout rates.
  2. Cost — tokens per request, cost per user and per completed task, cache hit rates.
  3. Quality — sampled LLM-as-judge scores, user feedback, task success, escalation rates.
  4. Safety — refusals, policy violations, prompt-injection attempts, PII leakage.

We go deeper into tooling in Observability and Evaluation for AI in Production.

The MLOps platform landscape

NeedOpen sourceManaged
Experiment tracking & registryMLflowWeights & Biases, SageMaker Model Registry, Vertex AI Model Registry
PipelinesKubeflow Pipelines, Argo Workflows, Airflow, RaySageMaker Pipelines, Vertex AI Pipelines, Azure ML
ServingvLLM, SGLang, KServe, Triton, Ray ServeBedrock, SageMaker endpoints, Vertex AI endpoints
LLM tracing & evalsLangfuse, Arize Phoenix, OpenTelemetryLangSmith, Braintrust, cloud-native options

There's no single correct stack. Small teams often do well with a managed model API, prompts in git, a lightweight eval harness in CI and an observability tool. Teams self-hosting open models on GPUs need the full platform — which is where Kubernetes comes in.