Why Enterprise Voice AI Still Relies on Cascaded Pipelines
Enterprise voice agents run on cascaded pipelines, not speech-to-speech models. Here is the pipeline engineering teams use in production.
⏭️ Forward Deployed: Voice AI on what works in 2026
With swyx, Basil Chatha, Varun Singh, Basia Sudol, Sudarshan Kamath, Steven Diaz, Tyler D'Silva · Sunday, August 30, 2026
In this Forward Deployed Engineering panel hosted by Basil Chatha, engineering leaders from Daily, Decagon, Vapi, Retell AI, and Smallest AI discuss the realities of building and deploying enterprise voice agents. The panel details why modular cascaded pipelines remain dominant over end-to-end speech-to-speech models, how engineering teams manage turn-taking and latency constraints, and the technical trade-offs between monolithic system prompts, specialized workflows, and self-hosted small language models.
Key takeaways
- Basia Sudol points out that enterprise voice agents require supervisor models to block prompt injection and social engineering attacks before an answer speaks. Read more →
- Outbound voice agents have asymmetric guardrail safety: in narrow tasks like debt collection, the agent can drop the call whenever a user veers off-script. Read more →
- Engineering teams spend months shaving 10 milliseconds off individual pipeline steps, but third-party backend APIs often cause massive 5-second bottlenecks. Read more →
- Large language models suffer from severe context degradation in voice calls, often retaining only the first 4% and last 4% of a prompt while ignoring instructions placed in the middle. Read more →
- Global voice models tuned for US English and Spanish fail immediately in markets like Japan and Arabic-speaking countries due to localized phonetic rules. Read more →
- OpenAI API latency spikes kill conversational voice agents, where delays over 500 milliseconds ruin human turn-taking. Read more →
6 articles from this episode
Stop Paying LLM Token Tax on 10-Second Voice Hangups
Tyler D'Silva and Varun Singh explain why monolithic prompts destroy voice agent unit economics on early hangups.
Fix Voice Agent Latency with Parallel Models and Fillers
Learn how voice AI engineers use parallel SLMs, waterfall text streams, and contextual fillers to mask multi-second API latency.
Why Voice AI Prompts Fail: The 4% Context Trap
Engineers from Vapi, Decagon, and Retell explain why monolithic voice agent prompts fail and when to switch to workflow graphs.
Why Monolithic Voice AI Fails in Tokyo and Riyadh
Global TTS engines break on local brand names and addresses. Vapi's Steven Diaz explains why modular voice pipelines beat end-to-end models.
Stop Routing Voice Agents Through Frontier LLM APIs
Fine-tuned small language models beat frontier APIs on voice latency and cost. Here is how engineers test and deploy SLMs for production voice agents.
More Latent Space episodes
- The Programmable Data Center — Evan Conrad, SF ComputeOct 25 · 6 articles
- AI in NYC: How Tenex Builds an AI-Native Engineering Team — Alex Lieberman & CJ HessOct 25 · 4 articles
- Robots That Predict the Future: Dyna-2 & Physical AI — Jason Ma, Dyna RoboticsOct 18 · 4 articles
- High Taste Testers: The AI Tools We Keep Coming Back ToOct 18 · 5 articles