How GPT-5, Claude, and Gemini are actually trained and served – Reiner Pope
Reiner Pope, CEO of MatX, breaks down the intricate details of how large language models like GPT-5, Claude, and Gemini are trained and served in cluster environments. He explains the critical role of batch size, mixture of experts, and parallelism in managing latency and cost, linking these technical elements to real-world AI API pricing structures. The discussion also ventures into the physical constraints of GPU rack design and the surprising architectural parallels between cryptographic protocols and neural networks.
- AI training and cryptographic protocols, despite aiming to extract vs. obscure structure, share a surprising architectural kinship in how they mix and scramble information. Read →
- A standard GPU rack, a few meters tall, typically houses around 64 GPUs, limited by power, weight, and cooling capacity. Read →
- Gemini 3.1's 50% price jump for context lengths over 200,000 tokens isn't arbitrary; it signals a hard constraint on memory bandwidth, not just compute, in underlying hardware. Read →
- MatX CEO Reiner Pope reveals how LLM batch size can slash inference costs 1000x—or spike latency. Learn the formula to balance speed and spend. Read →
- Pipeline parallelism is a strategy that slices an LLM vertically, allowing different layers to run on separate physical racks. This dramatically reduces the memory capacity needed per rack for storing model weights. Read →
- Expert Parallelism is King (for Racks): For deploying Mixture of Experts (MoE) layers in LLMs, the optimal strategy is “expert parallelism,” where different experts are mapped to different GPUs within a single, highly-connected rack. Read →