Home / Articles / Engineering
Engineering GPUs & Hardware

How Frontier Models Are Trained: Parallelism at Scale

Data, tensor, pipeline and expert parallelism — the distributed-systems engineering behind a single training run.

By SiliconMind EditorialNov 202615 min read
Key takeaways
  • A frontier training run is a distributed-systems problem spanning thousands to hundreds of thousands of accelerators.
  • Memory, not just compute, forces parallelism: weights, gradients and optimizer state for large models need terabytes.
  • Data, tensor, pipeline, sequence/context and expert parallelism are combined into "N-D" parallelism.
  • Mixed precision, activation checkpointing and sharded optimizers (ZeRO/FSDP) make models fit.
  • At scale, hardware failures are routine; checkpointing and fast recovery determine effective throughput.

Training a frontier language model is less like running a program and more like operating a power plant. Thousands of accelerators must work in lockstep for weeks or months, exchanging data constantly, while hardware fails, loss curves spike and engineers watch dashboards around the clock. This article explains the core techniques that make it possible.

The training pipeline at a glance

  1. Data — collect, filter, deduplicate and mix trillions of tokens from web text, code, books, papers and synthetic data. Data quality is one of the biggest levers on model quality.
  2. Tokenization — convert text into integer tokens using a learned vocabulary (often byte-pair encoding).
  3. Pre-training — train the model to predict the next token across the corpus. This is where most compute goes.
  4. Mid- and post-training — specialized data (long context, code, reasoning), supervised fine-tuning, and reinforcement learning from human and AI feedback or verifiable rewards.
  5. Evaluation and safety testing — capability benchmarks, red-teaming, and dangerous-capability evaluations.

Why one GPU isn't enough: the memory math

Consider a 70-billion-parameter dense model trained with the Adam optimizer in mixed precision. A common accounting is about 16 bytes per parameter:

ComponentBytes / parameter70B model
BF16 weights2140 GB
BF16 gradients2140 GB
FP32 master weights4280 GB
Adam moments (2 × FP32)8560 GB
Total (before activations)16~1.1 TB

That's before storing activations for the backward pass, which grow with batch size and sequence length. Even a GPU with nearly 200 GB of HBM can't hold this alone, so the model and its state must be split across many devices.

The parallelism toolbox

DATA PARALLEL fullmodel fullmodel batch Abatch B all-reduce grads TENSOR PARALLEL W[:, :k]W[:, k:] each layer splitacross GPUs (NVLink) PIPELINE PARALLEL GPU 0 · layers 1–20 GPU 1 · layers 21–40 GPU 2 · layers 41–60 micro-batches flowthrough stages Real runs combine all of them — plus sequence/context and expert parallelism
Fig. 1 — The three classic dimensions of parallelism.

Data parallelism

Each worker holds a copy of the model and processes a different slice of the batch. After the backward pass, gradients are averaged across workers with an all-reduce collective. Simple and efficient — but every worker needs the whole model, which doesn't fit for large models.

Sharded data parallelism (ZeRO / FSDP)

Microsoft's ZeRO technique, implemented in DeepSpeed and PyTorch's Fully Sharded Data Parallel (FSDP), shards optimizer state, gradients and even weights across data-parallel workers. Each worker gathers the parameters it needs just in time for each layer, then discards them. This removes the memory redundancy of plain data parallelism at the cost of more communication.

Tensor parallelism

Pioneered at scale in NVIDIA's Megatron-LM, tensor parallelism splits individual weight matrices across GPUs. For an MLP block, the first matrix can be split by columns and the second by rows so that only one all-reduce is needed per block. Communication is frequent, so tensor parallelism is usually kept inside a fast NVLink domain.

Pipeline parallelism

Layers are divided into stages on different devices, and the batch is split into micro-batches that flow through the pipeline. The challenge is the "bubble" — idle time while the pipeline fills and drains — reduced by schedules such as 1F1B and interleaved stages.

Sequence, context and expert parallelism

For long-context training, context parallelism (for example, ring attention) splits the sequence itself across devices. For Mixture-of-Experts models, expert parallelism places different experts on different GPUs and routes tokens between them with all-to-all communication.

Making it fit and run fast

  • Mixed precision — compute in BF16 or FP8 while keeping sensitive values in FP32.
  • Activation checkpointing — discard some activations during the forward pass and recompute them during backward, trading compute for memory.
  • Fused and IO-aware kernels — FlashAttention and fused layer operations reduce memory traffic.
  • Communication overlap — launch gradient reductions for one layer while computing the next.

The key efficiency metric is Model FLOPs Utilization (MFU): the fraction of the hardware's theoretical peak that goes into useful model math. Well-tuned large runs often achieve roughly 30–50% MFU; the rest is lost to communication, memory stalls and idle time.

Failures are the normal case

With tens of thousands of GPUs, something breaks constantly: a GPU throws memory errors, a network link flaps, a node overheats. Meta reported that during a 54-day Llama 3 training run on 16,384 H100 GPUs, it experienced hundreds of unexpected interruptions, most caused by hardware.

  • Frequent, fast checkpointing — save model and optimizer state often, asynchronously, to distributed storage.
  • Automated detection — health checks catch slow or faulty nodes ("stragglers") and evict them.
  • Hot spares — keep replacement nodes ready to minimize downtime.
  • Loss-spike handling — roll back to a checkpoint and skip problematic data batches when training becomes unstable.
Goodput vs throughputPeak throughput doesn't matter if the cluster spends hours recovering from failures. Labs increasingly track goodput — useful training progress per wall-clock hour — as the real measure of a cluster's effectiveness.

After pre-training: reinforcement learning at scale

Post-training has become a large share of compute. Reinforcement learning for reasoning and agentic skills requires generating many model outputs (rollouts), scoring them with reward models or verifiers (unit tests, math checkers, environment outcomes), and updating the policy. This mixes inference-heavy and training-heavy phases, and modern infrastructure is increasingly designed to do both efficiently on the same fleet.

Next: once a model is trained, serving it efficiently is its own discipline. Continue with The Economics of Inference.