Baseten's Long-Context LLMs: Cache, Disaggregate, Speculate
Founders, learn how Baseten efficiently handles 200,000-token LLM queries with cache-aware routing, prefill/decode disaggregation, and speculative decoding.
Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten
With swyx, Alessio Fanelli, Philip Kiely, Ali Taha · Sunday, August 9, 2026
This episode of Latent Space features Philip Kiely and Ali Taha from Baseten, discussing advanced inference engineering for AI models. They delve into strategies for optimizing model performance, reliability, and cost, particularly for large language models, and explore the future of hardware-software co-design. Key topics include managing long context queries, the complex process of supporting new model releases, novel quantization techniques, and the challenges of video generation.
Baseten's Proof: Quantize More, Get Better LLM Quality
Baseten's Ali Taha reveals how mathematically proven quantization can increase LLM throughput by 20% and improve fidelity. Stop trading quality for speed.
Rubin Era: AI Inference Shifts From Kernels to Orchestration
Under Nvidia's Rubin, AI inference engineering demands less low-level kernel tuning and more systems orchestration. Kiely and Taha debate the future.
Baseten's GLM 5.2 Rewrites Its Own GPU Kernels
Ali Taha from Baseten reveals how their GLM 5.2 model autonomously analyzes, writes, and implements its own optimized GPU kernels for faster inference.
Ali Taha: Auto-Regressive is the Future Bet for Long Video
Baseten's Ali Taha explains why diffusion models hit a quadratic bottleneck for long video and why auto-regressive architectures are the clear future.