Baseten's Long-Context LLMs: Cache, Disaggregate, Speculate
Founders, learn how Baseten efficiently handles 200,000-token LLM queries with cache-aware routing, prefill/decode disaggregation, and speculative decoding.
Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten
With swyx, Philip Kiely, Ali Taha · Sunday, August 9, 2026
This episode of Latent Space features Philip Kiely and Ali Taha from Baseten, discussing advanced inference engineering for AI models. They delve into strategies for optimizing model performance, reliability, and cost, particularly for large language models, and explore the future of hardware-software co-design. Key topics include managing long context queries, the complex process of supporting new model releases, novel quantization techniques, and the challenges of video generation.
Key takeaways
- To handle massive 200,000-token LLM queries cost-effectively, Baseten first checks if parts of the input have been seen before, using cache-aware routing to skip expensive re-computations. Read more →
- Conventional wisdom on quantization is dead: Baseten's research shows you can quantize more and still boost LLM quality. Read more →
- Nvidia's upcoming Rubin architecture signals a new era for AI inference, specifically designed with a deep understanding of large language model workloads from its inception. Read more →
- AI models are now getting good at autonomously optimizing their own underlying infrastructure. Baseten’s GLM 5.2 model proves this by rewriting its own GPU kernels. Read more →
- Current open-source video generation models, like 1.2.2, show a "night and day" quality gap compared to cutting-edge models like Kling or Veo, especially for long-form content. Read more →
5 articles from this episode
Baseten's Proof: Quantize More, Get Better LLM Quality
Baseten's Ali Taha reveals how mathematically proven quantization can increase LLM throughput by 20% and improve fidelity. Stop trading quality for speed.
Rubin Era: AI Inference Shifts From Kernels to Orchestration
Under Nvidia's Rubin, AI inference engineering demands less low-level kernel tuning and more systems orchestration. Kiely and Taha debate the future.
Baseten's GLM 5.2 Rewrites Its Own GPU Kernels
Ali Taha from Baseten reveals how their GLM 5.2 model autonomously analyzes, writes, and implements its own optimized GPU kernels for faster inference.
Ali Taha: Auto-Regressive is the Future Bet for Long Video
Baseten's Ali Taha explains why diffusion models hit a quadratic bottleneck for long video and why auto-regressive architectures are the clear future.
More Latent Space episodes
- The Future of Claude Code: Mods, Mutable Software, & Multiplayer Agents — Thariq Shihipar, AnthropicOct 4 · 4 articles
- Which GPU Clouds Are Actually Good? | ClusterMAX 3.0Oct 4 · 6 articles
- Recursive Language Models — Alex Zhang, MIT PhDOct 4 · 6 articles
- OpenAI’s New Agent Stack: Computer Use, Decisions API, UltraFast, Dots—Ari Weinstein & Nikunj HandaOct 4 · 6 articles