Home / Articles / MLOps
MLOps Cloud Engineering

Serving LLMs on Kubernetes: GPUs, vLLM and Autoscaling

A practical guide to self-hosting models on Kubernetes — GPU nodes, model servers, cold starts, autoscaling and smart routing.

By SiliconMind EditorialNov 202616 min read
Key takeaways
  • Kubernetes has become the standard control plane for self-hosted AI inference, but GPUs need extra components: drivers, the device plugin and monitoring.
  • The NVIDIA GPU Operator automates the GPU software stack on each node.
  • A model server such as vLLM runs as a Deployment exposing an OpenAI-compatible API; startup probes and model caching handle slow cold starts.
  • Autoscale on inference signals — queue depth, KV-cache usage — not CPU utilization.
  • Model-aware routing (Gateway API Inference Extension, llm-d) improves cache hits and latency at scale.

Calling a hosted model API is the fastest way to ship an AI feature. But many organizations also run models themselves — open-weight models for cost, data residency, customization or latency. When they do, the platform of choice is almost always Kubernetes. This guide walks through what it takes to serve LLMs on Kubernetes well, from GPU nodes to autoscaling.

The reference architecture

Clients Inference gatewayauth · routing · quotas KUBERNETES CLUSTER · GPU NODE POOL vLLM podGPU 0-1 (TP=2)/metrics · /health vLLM podGPU 2-3 (TP=2)/metrics · /health Prometheus + KEDA Model cache (PVC / NVMe) model-aware routing:queue depth · KV-cache hits
Fig. 1 — A typical self-hosted LLM serving stack: a gateway routes requests to model-server pods on GPU nodes, with metrics driving autoscaling.

Step 1 — GPU nodes and the GPU Operator

Kubernetes doesn't understand GPUs natively. A node needs the NVIDIA driver, the container toolkit (so containers can access the GPU) and the device plugin, which advertises GPUs to the scheduler as an extended resource called nvidia.com/gpu.

The NVIDIA GPU Operator installs and manages all of this as Kubernetes workloads: drivers, toolkit, device plugin, GPU Feature Discovery (which labels nodes with GPU model and memory), the DCGM exporter for metrics, and the MIG manager. Managed services such as Amazon EKS, Google GKE and Azure AKS offer GPU-optimized node images that handle part of this for you.

Best practices for GPU node pools:

  • Taint GPU nodes (e.g., nvidia.com/gpu=present:NoSchedule) so only workloads that tolerate the taint — your model servers — land on expensive hardware.
  • Separate pools by GPU type and select with node labels; a 70B model and an embedding model have very different needs.
  • Use node autoscaling (Karpenter on EKS, cluster autoscaler or node auto-provisioning elsewhere) to add GPU nodes on demand — and remove them when idle.

Step 2 — Deploy a model server

vLLM is the most common open-source choice: it implements continuous batching, PagedAttention, prefix caching and quantization, and exposes an OpenAI-compatible HTTP API. Here's a production-shaped Deployment for a 7B-class model on one GPU:

yaml
apiVersion: apps/v1 kind: Deployment metadata: name: llm-server labels: {app: llm-server} spec: replicas: 2 selector: matchLabels: {app: llm-server} template: metadata: labels: {app: llm-server} spec: tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule containers: - name: vllm image: vllm/vllm-openai:latest # pin an exact version tag in production command: ["vllm", "serve"] args: - "Qwen/Qwen2.5-7B-Instruct" - "--max-model-len=32768" - "--gpu-memory-utilization=0.90" - "--enable-prefix-caching" ports: - containerPort: 8000 env: - name: HF_TOKEN # only needed for gated models valueFrom: {secretKeyRef: {name: hf-token, key: token}} resources: limits: nvidia.com/gpu: 1 memory: 48Gi volumeMounts: - {name: model-cache, mountPath: /root/.cache/huggingface} - {name: shm, mountPath: /dev/shm} startupProbe: # model download + load can take minutes httpGet: {path: /health, port: 8000} periodSeconds: 10 failureThreshold: 90 readinessProbe: httpGet: {path: /health, port: 8000} periodSeconds: 5 volumes: - name: model-cache persistentVolumeClaim: {claimName: model-cache} - name: shm # shared memory for multi-GPU communication emptyDir: {medium: Memory, sizeLimit: 16Gi} --- apiVersion: v1 kind: Service metadata: name: llm-server spec: selector: {app: llm-server} ports: - port: 80 targetPort: 8000

Applications can now call http://llm-server/v1/chat/completions with any OpenAI-compatible client. For larger models, request multiple GPUs per pod and add --tensor-parallel-size; for models too large for one node, use multi-node serving patterns such as LeaderWorkerSet or Ray.

Step 3 — Tame cold starts

LLM pods are slow to start: pulling a multi-gigabyte image, downloading tens or hundreds of gigabytes of weights, loading them into GPU memory and compiling kernels. A pod that takes ten minutes to become ready makes autoscaling sluggish. Mitigations:

  • Cache weights on a shared volume, local NVMe or a pre-populated PVC rather than downloading from the internet on every start.
  • Stream weights directly from object storage into GPU memory using loaders designed for it.
  • Pre-pull images to GPU nodes and keep images lean.
  • Keep warm capacity — a minimum replica count sized for baseline traffic.
  • Use startup probes with generous thresholds so Kubernetes doesn't kill pods that are still loading.

Step 4 — Autoscale on the right signal

CPU utilization is meaningless for GPU inference. Scale on signals that reflect inference load. vLLM exposes Prometheus metrics such as the number of waiting requests and KV-cache usage. With KEDA, you can scale directly on a PromQL query:

yaml
apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: llm-server spec: scaleTargetRef: name: llm-server minReplicaCount: 1 maxReplicaCount: 8 cooldownPeriod: 600 # scale down slowly: new pods are expensive to start triggers: - type: prometheus metadata: serverAddress: http://prometheus.monitoring.svc:9090 query: sum(vllm:num_requests_waiting) threshold: "10"

Pod autoscaling and node autoscaling work together: KEDA adds replicas, the pending pods trigger the node autoscaler to provision GPU nodes, and scale-down returns them when traffic drops. Scale-to-zero is possible for dev or batch workloads, but cold starts make it a poor fit for interactive traffic.

Step 5 — Route requests intelligently

A standard round-robin load balancer treats every request and every pod as identical. LLM serving isn't like that: requests vary wildly in length, and a pod that already holds a conversation's prefix in its KV cache can answer much faster. Newer components address this:

  • Gateway API Inference Extension — a Kubernetes SIG project that adds model-aware routing to Gateway API implementations, picking endpoints based on queue length, KV-cache utilization and LoRA adapters loaded.
  • llm-d — an open-source project launched in 2025 by Red Hat, Google, IBM, NVIDIA, CoreWeave and others, combining vLLM with cache-aware routing and prefill/decode disaggregation on Kubernetes.
  • NVIDIA Dynamo — a distributed inference framework with KV-cache-aware routing and disaggregated serving across engines.
  • KServe — an open-source project providing an InferenceService abstraction, with LLM runtimes, autoscaling and canary rollouts built in.

Sharing GPUs

Not every model needs a whole GPU. Small embedding, reranking or classification models can share:

MethodIsolationUse when
MIG (Multi-Instance GPU)Hardware partitions with dedicated memory and compute (up to 7 per A100/H100-class GPU)Predictable performance for multiple small models
Time-slicingNone — processes take turnsDev/test, bursty low-priority workloads
MPSPartial — concurrent kernelsMany small inference processes
One server, many modelsApplication-levelMultiple LoRA adapters on a shared base model

Production checklist

  • Pin image and model versions; roll out changes with canaries.
  • Set request limits (max tokens, max concurrent requests) to protect latency for everyone.
  • Put authentication, rate limiting and usage accounting in the gateway.
  • Export GPU metrics (DCGM) and server metrics to Prometheus; alert on queue growth and error rates.
  • Use PodDisruptionBudgets so node upgrades don't take all replicas down at once.
  • Track cost per million tokens served — and compare against hosted APIs honestly.
Build vs buySelf-hosting pays off with steady high volume, strict data-residency needs or heavily customized models. For spiky or low-volume workloads, a managed API or platform such as Amazon Bedrock is usually cheaper once engineering time is counted. See The Economics of Inference.