Key Takeaways

  • Engineering teams spend months shaving 10 milliseconds off individual pipeline steps, but third-party backend APIs often cause massive 5-second bottlenecks.
  • Eliminating all silence makes voice agents feel rigid and robotic; real human speech relies on contextual fillers while thinking.
  • Steven Diaz recommends running small language models in parallel to classify user intent while firing audio fillers simultaneously.
  • Varun Singh uses waterfall execution where text output from a primary model forks directly into multiple downstream models at once.

Silence Breaks Voice UX Faster Than Delay

When developers build voice agents, their first instinct is to optimize for raw turn-taking speed. They want the agent to speak the millisecond the user stops talking. But conversational design in speech is counterintuitive: pure zero-latency responses feel unnatural, while dead air during database queries breaks trust instantly.

Basia Sudol explains that after months of work to “shave off like 10 milliseconds at a time across every single part of this pipeline,” the hardest failure mode remains customer backends. When an external API takes 5 seconds to look up an account balance, dead silence makes users think the call dropped.

Sudol notes that natural speech always includes thinking buffers: “It is normal for in normal conversation for humans to say a few words while they're thinking. And so we've actually found that when we have zero contextual fillers, sometimes it actually feels more rigid than when we don't.” Instead of hiding the wait, voice agents must handle it explicitly. Saying, “Give me a sec, looking at that,” immediately confirms receipt and buys the backend the five seconds it needs to resolve.

Parallelize Inference Instead of Waiting on the Main Loop

Masking latency with fillers only works if your agent triggers them without delaying the main response. If your system prompt runs a heavy 70B model sequentially before deciding whether to speak or call a tool, your filler itself arrives two seconds late.

Steven Diaz points to parallel small language models as the fix. By streaming transcription output directly to a tiny classifier model, the system identifies the user's intent in tens of milliseconds: “What if you have streaming down to a small language model which can do inference in a very little time? So think about classifiers to maybe collect the intent while the pipeline is still maybe adding a filler word as well to keep a consistent experience.”

Varun Singh takes parallel execution a step further by waterfalling outputs across specialized models. Instead of waiting for a complete generation to finish before triggering downstream logic, the text stream forks in real time: “The text output from the first forks into multiple LLMs you can think of it as like a waterfall.” This architecture lets you run validation, tool calling, and speech synthesis concurrently rather than locking the thread on a single monolithic prompt.

What to Do With This

Audit your voice pipeline's longest tool call tomorrow. If your backend lookup exceeds 800 milliseconds, insert an immediate filler phrase trigger running on an edge classifier before the primary model finishes its tool execution. Stop trying to hit zero-millisecond turnarounds on complex queries; add natural conversational bridges instead.