Crypto's Feistel Networks Slash AI Training Memory Costs
Reiner Pope reveals how cryptographic Feistel networks led to RevNets, cutting neural network memory use during training. Founders, see how architectural cross-pollination saves real costs.
How GPT-5, Claude, and Gemini are actually trained and served – Reiner Pope
With Dwarkesh Patel, Reiner Pope · Sunday, April 26, 2026
Reiner Pope, CEO of MatX, breaks down the intricate details of how large language models like GPT-5, Claude, and Gemini are trained and served in cluster environments. He explains the critical role of batch size, mixture of experts, and parallelism in managing latency and cost, linking these technical elements to real-world AI API pricing structures. The discussion also ventures into the physical constraints of GPU rack design and the surprising architectural parallels between cryptographic protocols and neural networks.
Key takeaways
- AI training and cryptographic protocols, despite aiming to extract vs. obscure structure, share a surprising architectural kinship in how they mix and scramble information. Read more →
- A standard GPU rack, a few meters tall, typically houses around 64 GPUs, limited by power, weight, and cooling capacity. Read more →
- Gemini 3.1's 50% price jump for context lengths over 200,000 tokens isn't arbitrary; it signals a hard constraint on memory bandwidth, not just compute, in underlying hardware. Read more →
- Pipeline parallelism is a strategy that slices an LLM vertically, allowing different layers to run on separate physical racks. This dramatically reduces the memory capacity needed per rack for storing model weights. Read more →
- Expert Parallelism is King (for Racks): For deploying Mixture of Experts (MoE) layers in LLMs, the optimal strategy is “expert parallelism,” where different experts are mapped to different GPUs within a single, highly-connected rack. Read more →
6 articles from this episode
GPU Racks: The Hidden 8x Bottleneck in Your LLM
MatX CEO Reiner Pope reveals the 8x slower inter-rack network speed bottlenecking LLMs. Understand how GPU rack design limits MoE scale-up and what to do.
LLM API Costs: Context Length Reveals Memory Bottlenecks
Reiner Pope decodes LLM API pricing, showing how context length and input/output costs expose underlying hardware limits and memory tiers.
LLM Batch Size: 1000x Cheaper Inference or Crushing Latency?
MatX CEO Reiner Pope reveals how LLM batch size can slash inference costs 1000x—or spike latency. Learn the formula to balance speed and spend.
Pope: Pipelining Cuts LLM Weights, Not KV Cache Memory
MatX CEO Reiner Pope unpacks pipeline parallelism for LLMs. It slashes model weight memory but hits KV cache limits, making inference a 'no-brainer' but training a hard trade-off.
LLM MoE Scaling: Your GPU Rack is The Hard Limit
Reiner Pope of MatX reveals how Mixture of Experts (MoE) layers scale within a GPU rack but bottleneck across racks for LLMs.
More Dwarkesh Podcast episodes
- AI researchers debate how close we are to recursive self-improvementSep 13 · 6 articles
- Ajeya Cotra – Inside the OpenAI agent swarm that hacked Hugging FaceSep 6 · 4 articles
- The OpenAI/Hugging Face attack, clearly explainedSep 6 · 5 articles
- Ryan Greenblatt – What happens once AI can automate AI research?Aug 16 · 3 articles