The Economics of Inference: KV Caches, Batching and Quantization
Training makes headlines; inference pays the bills. The techniques that make tokens cheap and fast.
Key takeaways
- Inference has two phases: compute-bound prefill (reading the prompt) and memory-bound decode (generating tokens one at a time).
- The KV cache avoids recomputing attention over past tokens but can consume more memory than the weights at long context.
- Continuous batching and PagedAttention keep GPUs busy and memory efficiently used.
- Quantization, speculative decoding and prompt caching cut cost and latency dramatically.
- Measure cost per completed task, time to first token and tokens per second — not just price per million tokens.
Training a frontier model is a one-time (if enormous) expense. Inference — running the model for every user request — is a recurring cost that grows with adoption. As agents generate millions of tokens per task, inference efficiency has become the core economic problem of the AI industry. This article explains where the cost comes from and the techniques used to drive it down.
Two phases: prefill and decode
Prefill processes the entire prompt at once. It involves large matrix multiplications with high arithmetic intensity, so it is compute-bound, and it determines time to first token (TTFT).
Decode generates one token at a time. Each step must read every model weight from memory but does only about two FLOPs per weight per sequence. At small batch sizes it is memory-bandwidth-bound and determines output tokens per second.
A back-of-envelope ceiling
A 70B-parameter model in BF16 occupies about 140 GB. On a GPU with ~3.35 TB/s of memory bandwidth, reading all weights once takes about 140 / 3350 ≈ 42 ms — so a single sequence on a single such GPU tops out around 24 tokens per second, no matter how many FLOPS the chip has. (In practice the model would be split across several GPUs, which multiplies available bandwidth.) The way to use the idle compute is to process many sequences per weight read: batching.
The KV cache
In attention, each new token attends to the keys (K) and values (V) of every previous token. Recomputing them at each step would be wasteful, so servers store them in a KV cache. The cost is memory:
# KV cache bytes per token
# = 2 (K and V) x layers x kv_heads x head_dim x bytes_per_value
layers, kv_heads, head_dim, bytes_per_value = 80, 8, 128, 2 # a 70B-class model with GQA, BF16
per_token = 2 * layers * kv_heads * head_dim * bytes_per_value
print(per_token / 1024, "KiB per token") # 320 KiB
print(per_token * 128_000 / 1e9, "GB for a 128K context") # ~42 GB
For long contexts and many concurrent users, the KV cache can exceed the size of the weights. That's why architectures use grouped-query attention (GQA) or multi-head latent attention (MLA) to shrink it, and why serving systems quantize or offload the cache.
Batching and memory management
- Continuous (in-flight) batching — instead of waiting for a whole batch to finish, the scheduler adds new requests and removes completed ones at every decode step, keeping the GPU saturated.
- PagedAttention — introduced by the vLLM project, it stores the KV cache in fixed-size blocks like virtual memory pages, eliminating fragmentation and enabling sharing of common prefixes.
- Prefill/decode disaggregation — run the two phases on separate GPU pools tuned for each, transferring the KV cache between them. NVIDIA Dynamo, vLLM and SGLang support variants of this.
Quantization
Storing weights in fewer bits reduces memory footprint and the bytes read per decode step — directly increasing tokens per second in the memory-bound regime.
| Technique | What's quantized | Notes |
|---|---|---|
| FP8 | Weights + activations | Near-lossless for most models on Hopper/Blackwell |
| INT4 weight-only (GPTQ, AWQ) | Weights | Popular for self-hosted open models |
| FP4 (NVFP4, MXFP4) | Weights + activations | Blackwell-class hardware; block scaling preserves accuracy |
| KV cache quantization | K and V tensors | Shrinks long-context memory |
Speculative decoding
A small, fast draft model (or extra prediction heads) proposes several tokens ahead; the large model verifies them all in one parallel forward pass and accepts the longest correct prefix. Because verification is parallel like prefill, accepted tokens come almost for free. Output is identical to the large model's distribution, and speedups of 2–3× are common for predictable text such as code.
Prompt caching
Many requests share long prefixes: a system prompt, tool definitions, a document, or an agent's growing conversation history. Prompt caching stores the computed KV cache for that prefix and reuses it on later requests. API providers pass the savings on — cached input tokens are typically billed at around a tenth of the normal input price — and latency drops because prefill is skipped.
Batch APIs and service tiers
If results aren't needed immediately — evaluations, document processing, data labeling — batch APIs process requests asynchronously, typically at around half the standard price. Some providers also offer premium "fast" modes that deliver higher output speed for a higher per-token price.
The metrics that matter
- TTFT — time to first token; drives perceived responsiveness.
- Output tokens per second per user — drives how quickly long answers stream.
- Throughput — total tokens per second per GPU; drives provider cost.
- Cost per completed task — the number that matters for products. An agent that finishes in fewer steps with a stronger model can be cheaper overall than a cheap model that loops.
Infrastructure economics are covered further in AI Data Centers, and hardware fundamentals in Why AI Runs on GPUs.