AMD MI300X benchmarks within 10–20% of the NVIDIA H200 on most AI training tasks and costs roughly half as much per chip. NVIDIA still owns over 80% of AI training deployments. The gap is not the hardware — it's a decade and a half of software compounding.
This distinction matters enormously for anyone buying, renting, or investing in AI infrastructure. NVIDIA's valuation is not justified by GPU specs alone — it's justified by CUDA lock-in and the economic inertia of an ecosystem that every framework, library, and model was built on first.
AMD vs NVIDIA AI Training: The Hardware Specs
On raw silicon, AMD has made a real run. The MI300X is not a token competitor — it is a serious chip with specifications that beat NVIDIA in several categories:
| Spec | AMD MI300X | NVIDIA H200 | NVIDIA B200 |
|---|---|---|---|
| HBM Memory | 192 GB HBM3 | 141 GB HBM3e | 192 GB HBM3e |
| Memory Bandwidth | 5.3 TB/s | 4.8 TB/s | 8.0 TB/s |
| FP8 FLOPS (Peak) | 2,610 TFLOPS | 3,958 TFLOPS | 9,000 TFLOPS |
| BF16 FLOPS | 1,307 TFLOPS | 1,979 TFLOPS | 4,500 TFLOPS |
| TDP | 750W | 700W | 1,000W |
| Street Price (est.) | $15–20K | $30–40K | $40–50K+ |
| Interconnect | Infinity Fabric (64 GB/s) | NVLink 4.0 (900 GB/s) | NVLink 5.0 (1.8 TB/s) |
Sources: AMD/NVIDIA spec sheets; street prices are estimates and vary by contract volume.
The CUDA Moat: Why Hardware Specs Don't Tell the Full Story
CUDA launched in 2006. For 15+ years, every serious ML framework — PyTorch, TensorFlow, JAX — optimized for CUDA first and everything else later. The result is an ecosystem of thousands of libraries, kernels, and optimizations that simply do not exist in equivalent form for AMD ROCm.
cuDNN
NVIDIA's deep learning library, hand-tuned for convolutions and attention. ROCm's MIOpen is improving but still trails on some transformer kernels.
NCCL / NVLink
NVIDIA's collective communications library is tightly coupled to NVLink hardware. AMD's RCCL over Infinity Fabric achieves only ~7% of NVLink 4.0's bandwidth at scale.
FlashAttention
The attention kernel that powers most frontier model training was written for CUDA. AMD port exists but lags on optimization updates by months.
Triton
OpenAI's GPU kernel language defaults to CUDA. AMD support was added but community adoption and kernel libraries are still NVIDIA-dominant.
The engineering cost of porting a training stack to ROCm is real. Teams report 4–12 weeks of debugging and performance tuning before matching CUDA throughput on AMD hardware. That engineering time alone can exceed the hardware cost savings for smaller runs.
Where AMD Is Actually Winning: Inference
The MI300X's 192 GB of HBM makes it genuinely better than H200 for inference on large models that don't fit comfortably into 80 GB or 141 GB. LLM inference is often memory-bandwidth-bound rather than compute-bound — which plays to AMD's architectural strength.
Deployed MI300X for inference on select Azure OpenAI endpoints — announced at Ignite 2024 as a cost-efficient alternative to H100 for serving requests.
Runs Llama 3 inference on AMD MI300X clusters. Has publicly stated AMD is part of its multi-vendor GPU strategy to reduce NVIDIA dependency.
Added AMD Instinct instances (A3 Mega with MI300X) to Google Cloud catalog as lower-cost alternative to NVIDIA H100/H200 for inference serving.
None of these deployments represent AMD winning the foundation model training market. They represent AMD winning the inference cost battle at scale — which is still a multi-billion-dollar market, just a different one.
The Blackwell Problem: AMD Closed the H100 Gap, NVIDIA Launched B200
By mid-2025, AMD had largely closed the H100 performance gap with MI300X. The problem: NVIDIA launched Blackwell (B200 and GB200 NVL72 rack) simultaneously, which extended the performance lead before AMD could capitalize on parity.
B200 delivers roughly 2.5x the FP8 throughput of H200 and benefits from NVLink 5.0 at 1.8 TB/s — almost double NVLink 4.0. AMD's CDNA4 (MI350X) is expected in late 2026 and will target GB200 performance, but it will again be launching into a market where NVIDIA's next generation (Rubin) is already on the roadmap.
This is the structural dynamic AMD faces: it can only close the gap to NVIDIA's current generation while NVIDIA ships the next one. The software moat means even parity does not translate to market share in training.
What This Means for AI Infrastructure Buyers in 2026
Use AMD MI300X When:
- ✓ Running inference on large models (70B+ parameters)
- ✓ Memory-constrained workloads needing 192 GB per chip
- ✓ Budget-constrained startups with engineering bandwidth for ROCm
- ✓ You can accept some tooling friction for 40–50% hardware cost reduction
- ✓ Running workloads where ROCm support is already mature (fine-tuning Llama)
Stick With NVIDIA When:
- ✓ Pre-training frontier models at scale (billions of GPU-hours)
- ✓ Your team runs custom CUDA kernels or needs cutting-edge attention implementations
- ✓ You need NVLink bandwidth for large-scale distributed training
- ✓ Engineering time is more expensive than hardware cost
- ✓ You need access to the widest cloud spot market (H100s are everywhere)
The Investor View: NVIDIA's Moat Is Wider Than the Market Realizes
I track AI valuations and capex cycles on the Big Tech Earnings dashboard. The story on AMD vs NVIDIA is not fundamentally a hardware race — it is a platform lock-in story that most infrastructure investors underweight.
NVIDIA's $3T+ market cap is partly justified by GPU scarcity and AI capex tailwinds, but the durable part of that valuation is the CUDA ecosystem. AMD could ship a chip twice as fast as H200 tomorrow and still face 18–24 months of ecosystem migration friction before winning meaningful training market share.
The scenario where AMD wins is not hardware parity — it is ROCm reaching CUDA compatibility across the entire toolchain, which would require AMD to out-execute NVIDIA on software for 5+ consecutive years. AMD has shown real progress ($100M+ ROCm engineering investment since 2023), but the gap in software quality remains the dominant variable.
AMD wins on specs and price. NVIDIA wins on ecosystem and distribution.
Until ROCm matches CUDA across the full ML stack, NVIDIA's training monopoly survives regardless of how good the AMD hardware gets.
Track AI chip spending and hyperscaler capex on the Big Tech Earnings Dashboard at Value Add VC. Originally published in the Trace Cohen newsletter.