Home / Articles / MLOps
MLOps Engineering GPUs & Hardware

Inference Servers Compared: vLLM, SGLang, TensorRT-LLM and More

The engine you serve a model with can change throughput several-fold. How the major inference servers differ and how to benchmark them.

By SiliconMind EditorialNov 202612 min read
Key takeaways
  • The inference server determines throughput, latency and cost more than almost any other deployment choice.
  • vLLM is the broad default; SGLang excels at prefix-heavy and structured workloads; TensorRT-LLM squeezes the most out of NVIDIA hardware.
  • Triton and NVIDIA Dynamo orchestrate engines at scale; llama.cpp and Ollama target local and edge deployment.
  • All major servers now share core techniques: continuous batching, paged KV cache, prefix caching, quantization and speculative decoding.
  • Benchmark on your own traffic shape — input/output lengths and concurrency — and measure at your latency target.

Once you've chosen a model, the next decision that most affects cost and user experience is the inference server — the software that loads the weights, schedules requests onto the GPU and streams tokens back. The same model on the same GPU can differ several-fold in throughput depending on the engine and its configuration. Here is a practical guide to the main options.

What an inference server does

  • Scheduling — continuous batching of many concurrent requests at every decode step.
  • Memory management — paging the KV cache, evicting and reusing cached prefixes.
  • Kernels — fast attention, fused operations and quantized matrix math for specific GPUs.
  • Parallelism — splitting large models across GPUs and nodes.
  • API surface — usually an OpenAI-compatible HTTP API, plus metrics and health endpoints.

If these concepts are new, start with The Economics of Inference.

The contenders

vLLM

Born at UC Berkeley, where its authors introduced PagedAttention, vLLM is now the most widely used open-source LLM server and a PyTorch Foundation project. It supports a huge range of model architectures and hardware (NVIDIA, AMD, Google TPU, AWS Neuron and more), with continuous batching, prefix caching, many quantization formats, speculative decoding, multi-LoRA serving and tensor, pipeline and expert parallelism. Its breadth and community make it the sensible default.

SGLang

SGLang, from the LMSYS community, introduced RadixAttention: it keeps the KV cache in a radix tree so any shared prefix — system prompts, few-shot examples, multi-turn chats, agent loops — is reused automatically. It also has a fast engine for constrained decoding (JSON and grammar-guided output) and strong large-scale MoE serving. It shines in agentic and prefix-heavy workloads.

TensorRT-LLM

NVIDIA's library for maximum performance on NVIDIA GPUs. It provides highly optimized kernels, in-flight batching, paged KV cache and first-class support for low-precision formats such as FP8 and FP4 on recent hardware. It traditionally required compiling model-specific engines; newer releases add a PyTorch-based workflow that's easier to use. Choose it when you're all-in on NVIDIA and want peak tokens per GPU.

Triton Inference Server and NVIDIA Dynamo

Triton is a general-purpose model server supporting many backends (TensorRT-LLM, vLLM, PyTorch, ONNX, Python) with dynamic batching and ensembles — useful when serving LLMs alongside vision, speech or classic ML models. Dynamo, NVIDIA's newer distributed inference framework, coordinates engines such as vLLM, SGLang and TensorRT-LLM across many GPUs with disaggregated prefill/decode and KV-cache-aware routing.

Local and edge: llama.cpp and Ollama

llama.cpp runs quantized models (GGUF format) efficiently on CPUs, Apple Silicon and consumer GPUs. Ollama wraps it in a friendly CLI and local API. These are ideal for laptops, on-device apps and air-gapped environments — not for high-concurrency data center serving.

Others worth knowing

LMDeploy (strong quantized inference), Hugging Face's serving tools, Ray Serve (Python-native serving and composition, often wrapping vLLM), and KServe (Kubernetes-native model serving abstraction) all appear frequently in production stacks.

Comparison at a glance

EngineStrengthsTrade-offsBest for
vLLMBroad model and hardware support, huge community, OpenAI-compatible APIPeak performance on a given GPU may trail specialized enginesDefault choice for most teams
SGLangAutomatic prefix reuse, fast structured output, large MoE servingSmaller ecosystem than vLLMAgents, multi-turn chat, JSON-heavy workloads
TensorRT-LLMTop NVIDIA performance, FP8/FP4 kernelsNVIDIA-only; more tuning effortHigh-volume serving on NVIDIA fleets
Triton / DynamoMulti-model orchestration, distributed servingMore moving partsLarge platforms, mixed model types
llama.cpp / OllamaRuns anywhere, simpleLimited concurrencyLocal, edge, development

The techniques have converged

A few years ago engines differed in which techniques they supported. Today, all serious engines implement continuous batching, paged KV caches, prefix caching, quantization, speculative decoding and tensor parallelism. The differences are now in how well they're implemented for particular models, hardware and traffic patterns — which is why benchmarking matters more than feature lists.

How to benchmark properly

  1. Match your traffic shape. Long inputs with short outputs (RAG, classification) stress prefill; short inputs with long outputs (generation, reasoning) stress decode. Use realistic length distributions, not fixed values.
  2. Sweep concurrency. Measure from 1 to hundreds of concurrent requests and plot throughput against latency.
  3. Fix a latency target. The meaningful number is the throughput you can sustain while meeting your SLO — for example, p95 time to first token under 1 second and at least 30 output tokens per second per user.
  4. Include prefix reuse if your production traffic has it; it can change the winner.
  5. Check quality after quantization with your own evals.
  6. Compute cost per million tokens at that SLO using your actual GPU price.
bash
# Example: benchmark an OpenAI-compatible endpoint with vLLM's built-in tool vllm bench serve \ --backend openai-chat --endpoint /v1/chat/completions \ --base-url http://llm-server \ --model Qwen/Qwen2.5-7B-Instruct \ --dataset-name random --random-input-len 2000 --random-output-len 300 \ --num-prompts 500 --max-concurrency 64
Watch for misleading chartsVendor benchmarks usually report peak throughput at very high concurrency, where per-user latency may be unacceptable. Always ask what the latency was at the reported throughput.

Recommendation

Start with vLLM for its breadth and community. Evaluate SGLang if your workload is agentic or prefix-heavy, and TensorRT-LLM or Dynamo when you operate large NVIDIA fleets and every percent of throughput matters. Whatever you choose, deploy it behind a stable OpenAI-compatible interface so you can switch engines later — see Serving LLMs on Kubernetes.