NVIDIA's Full-Stack Bet: CUDA, NVLink and the AI Factory
NVIDIA doesn't just sell chips. It sells a vertically integrated platform — and that is the real moat.
Key takeaways
- NVIDIA's advantage is a full stack: chips, systems, networking and a software ecosystem built over nearly two decades.
- CUDA (2007) created a developer base and library ecosystem that competitors still work to match.
- The 2020 Mellanox acquisition gave NVIDIA InfiniBand and high-performance Ethernet — the networking half of AI clusters.
- Hopper introduced FP8 training; Blackwell moved the unit of compute to the rack with NVL72.
- NVIDIA now pitches "AI factories": data centers whose product is tokens.
It is tempting to describe NVIDIA as a chip company. That undersells what it has built. NVIDIA sells silicon, but also servers, racks, network switches, cables, compilers, math libraries, inference microservices and cloud capacity. Each layer reinforces the others, and together they form one of the strongest platform positions in technology.
From graphics to general-purpose compute
NVIDIA spent its first decade building graphics processors for PC gaming. The pivotal decision came in 2006–2007 with CUDA, a programming model that let developers write general-purpose code for the GPU in a C-like language. For years it served a relatively niche market of scientific computing and finance.
That changed in 2012 when AlexNet, trained on two NVIDIA consumer GPUs, won the ImageNet competition. Deep learning researchers adopted CUDA en masse, and NVIDIA began designing data center products specifically for AI.
The data center GPU lineage
| Architecture | Flagship | Notable for AI |
|---|---|---|
| Pascal (2016) | P100 | First HBM2 and NVLink generation |
| Volta (2017) | V100 | Introduced Tensor Cores |
| Ampere (2020) | A100 | TF32/BF16, structured sparsity, Multi-Instance GPU |
| Hopper (2022) | H100 / H200 | FP8 Transformer Engine; H200 added larger, faster HBM3e |
| Blackwell (2024) | B200 / GB200 | Two-die design, FP4, 5th-gen NVLink, rack-scale NVL72 |
| Rubin (announced) | Vera Rubin | Next platform on NVIDIA's annual cadence, paired with the Vera CPU |
NVIDIA moved to a roughly annual release cadence, with "Ultra" refreshes between major architectures. For customers this means a steady stream of performance-per-watt improvements; for competitors it means a fast-moving target.
CUDA: the real moat
Hardware can be matched; ecosystems are harder. Nearly two decades of CUDA development produced:
- Millions of developers who know the programming model.
- Highly tuned libraries — cuBLAS, cuDNN, CUTLASS, NCCL — that frameworks depend on.
- First-class support in PyTorch, JAX and virtually every AI research codebase.
- Inference stacks such as TensorRT-LLM and the Dynamo serving framework, plus prepackaged NIM microservices that bundle models with optimized runtimes.
When a new model architecture appears, it usually runs well on NVIDIA hardware first. That time-to-performance advantage matters enormously in a field that changes monthly.
Networking: the underrated half
In 2020 NVIDIA completed its acquisition of Mellanox, the leading maker of InfiniBand networking. In hindsight it was one of the most strategic acquisitions of the era. Training a frontier model means thousands of GPUs exchanging gradients constantly; network bandwidth and latency directly determine how much of the cluster's compute is actually used.
NVIDIA now offers InfiniBand (Quantum switches), an AI-tuned Ethernet platform (Spectrum-X), BlueField DPUs for offloading networking and security, and NVLink for scale-up. It has also invested in co-packaged optics to cut the power used by optical transceivers in very large clusters.
Blackwell and the rack as the computer
With Blackwell, NVIDIA shifted the unit of design from the chip to the rack. The GB200 NVL72 combines 36 Grace CPUs and 72 Blackwell GPUs connected by NVLink switches into a single, liquid-cooled rack that software can treat as one giant GPU domain. The NVLink "spine" uses copper cabling inside the rack to avoid the cost and power of optics.
This matters most for inference of very large mixture-of-experts models, where experts are spread across many GPUs and tokens must be routed between them at high speed.
The "AI factory" pitch
NVIDIA's CEO Jensen Huang frames modern data centers as AI factories: facilities that take in electricity and data and produce tokens. The framing is deliberate. If the output is tokens, the key metrics are tokens per second, tokens per watt and cost per token — and NVIDIA argues that a faster, more integrated platform lowers the cost per token even when the hardware is expensive.
Competitive pressure
NVIDIA's dominance has invited competition from several directions:
- AMD Instinct GPUs offer large HBM capacity and an increasingly mature ROCm software stack.
- Hyperscaler silicon — Google TPU, AWS Trainium, Microsoft Maia, Meta MTIA — targets internal workloads and selected customers.
- Startups such as Cerebras, Groq and others pursue radically different architectures, often focused on low-latency inference.
- Software portability efforts like Triton and higher-level compilers reduce dependence on hand-written CUDA kernels.
Even so, most frontier labs still train primarily on NVIDIA hardware, and demand has repeatedly exceeded supply. The more realistic question for the next few years is not whether NVIDIA is displaced, but how much share specialized and in-house accelerators capture as inference becomes the dominant workload.
What to watch
- HBM supply and pricing — a key constraint on how many GPUs can ship.
- Adoption of rack-scale systems and liquid cooling across cloud providers.
- Inference share captured by custom silicon versus GPUs.
- Export controls and geopolitics affecting where advanced chips can be sold.