Why AI Runs on GPUs: A Hardware Deep Dive
Tensor Cores, HBM, memory bandwidth and the arithmetic of matrix multiplication — what actually makes a GPU fast at deep learning.
Key takeaways
- Neural networks are dominated by matrix multiplication, which is massively parallel — exactly what GPUs are built for.
- Tensor Cores execute small matrix multiply-accumulate operations in hardware and power most AI FLOPS.
- Lower-precision formats (BF16, FP8, FP4) multiply throughput and reduce memory traffic.
- For LLM inference, memory bandwidth — not raw FLOPS — is often the limiting factor, which is why HBM matters so much.
- At cluster scale, interconnect bandwidth (NVLink, InfiniBand) is as important as the chip itself.
Ask why NVIDIA became one of the most valuable companies in the world and the short answer is "GPUs." The longer answer is more interesting: deep learning happens to reduce almost entirely to one mathematical operation, and GPUs happen to be extraordinarily good at it. This deep dive walks through the hardware from the arithmetic up.
It's all matrix multiplication
A Transformer layer is mostly a sequence of matrix multiplies (GEMMs): projecting tokens into queries, keys and values; computing attention scores; and the large feed-forward (MLP) blocks. For a model with N parameters, a commonly used rule of thumb is that a forward pass costs about 2N floating-point operations per token, and training (forward plus backward) about 6N per token.
Multiply 6N by the trillions of tokens in a pre-training dataset and you get numbers on the order of 1024–1026 operations for frontier models. The only practical way to perform that much arithmetic is to do enormous amounts of it in parallel.
CPU vs GPU: latency vs throughput
A CPU is designed to run a small number of threads as fast as possible, with large caches, branch predictors and out-of-order execution. A GPU makes the opposite trade: thousands of simpler cores, organized to run the same instruction across many data elements at once, hiding memory latency by switching between huge numbers of threads.
Inside a modern data center GPU
An NVIDIA data center GPU such as the H100 (Hopper) or B200 (Blackwell) is built from many Streaming Multiprocessors (SMs). Each SM contains:
- CUDA cores for general floating-point and integer math.
- Tensor Cores — specialized units that compute a small matrix multiply-accumulate (D = A×B + C) in a single operation. These deliver the overwhelming majority of the chip's AI throughput.
- Registers and shared memory — fast on-chip storage that kernels use to reuse data and avoid trips to main memory.
Tensor Cores first appeared in the Volta generation (V100, 2017). Each generation since has added faster units and new number formats: TF32 and BF16 in Ampere, FP8 with the "Transformer Engine" in Hopper, and FP4 support in Blackwell.
Precision: fewer bits, more math
Neural networks tolerate reduced numerical precision surprisingly well. Halving the bits per number roughly doubles Tensor Core throughput and halves the memory needed to store and move each value.
| Format | Bits | Typical use |
|---|---|---|
| FP32 | 32 | Master weights, optimizer state, sensitive reductions |
| BF16 / FP16 | 16 | Standard mixed-precision training |
| FP8 | 8 | Training and inference on Hopper and later, with per-tensor or per-block scaling |
| INT8 / INT4 | 8 / 4 | Quantized inference (weights) |
| FP4 (e.g., NVFP4, MXFP4) | 4 | High-throughput inference on Blackwell-class hardware |
The trick is scaling: low-precision formats have tiny dynamic range, so values are grouped into blocks that share a scale factor. Getting this right is a major part of modern kernel engineering.
The memory wall and HBM
Raw FLOPS are only useful if data can reach the Tensor Cores fast enough. That's why AI GPUs use High Bandwidth Memory (HBM): stacks of DRAM dies placed right next to the GPU on a silicon interposer, connected through thousands of wires. An H100 SXM provides 80 GB of HBM3 at more than 3 TB/s; Blackwell-generation parts roughly double the bandwidth and more than double capacity.
A useful mental model is arithmetic intensity — FLOPs performed per byte moved. If a workload does fewer operations per byte than the chip's ratio of compute to bandwidth, it is memory-bound and extra FLOPS sit idle.
Software: the CUDA moat
Hardware alone doesn't explain NVIDIA's position. CUDA, plus libraries like cuBLAS (linear algebra), cuDNN (neural network primitives) and NCCL (multi-GPU communication), means PyTorch and JAX run fast on NVIDIA GPUs out of the box. Highly optimized kernels like FlashAttention — which computes attention in tiles kept in on-chip memory, avoiding materializing the full attention matrix — were written first for CUDA.
Competitors such as AMD (ROCm), Google (XLA on TPUs) and AWS (the Neuron SDK for Trainium) have invested heavily to close this gap, and compilers like OpenAI's Triton make it easier to write portable high-performance kernels.
Scaling beyond one chip
No single GPU can hold or train a frontier model. Chips are connected at two levels:
- Scale-up: NVLink and NVSwitch connect GPUs within a server or rack with very high bandwidth, so they can act like one large accelerator. Blackwell's GB200 NVL72 links 72 GPUs in a single NVLink domain.
- Scale-out: InfiniBand or high-performance Ethernet connects thousands of servers into a cluster for training.
How work is split across those links — data, tensor, pipeline and expert parallelism — is covered in How Frontier Models Are Trained.
The alternatives
GPUs are general-purpose parallel processors. Custom accelerators trade flexibility for efficiency on specific workloads: Google's TPUs use large systolic arrays for matrix math; AWS Trainium and Inferentia target training and inference on AWS; Microsoft's Maia and Meta's MTIA serve their own fleets. AMD's Instinct GPUs compete directly with NVIDIA. The common thread: every design is organized around feeding matrix units with enough memory bandwidth.