Key Takeaways

  • Large language models suffer from severe context degradation in voice calls, often retaining only the first 4% and last 4% of a prompt while ignoring instructions placed in the middle.
  • Monolithic system prompts degrade rapidly in multi-turn voice sessions, sometimes going off the rails after 10,000 tokens without aggressive context compaction.
  • Steven Diaz from Vapi recommends offloading instructions and splitting 10 to 15 step workflows into separate sub-agent nodes to prevent skipped steps.
  • Tyler D'Silva notes that graph-based nodes work best when conversation paths are predictable, while open-ended inbound calls demand situational rule injection.
  • Basia Sudol explains that Decagon dynamically inspects user state and injects rules only when relevant, keeping context windows small and responses fast.

The Disagreement

Every team building voice agents runs into the same wall: should you pack all your business logic into one large prompt, or should you split the call into a graph of stateful nodes?

Advocates of the monolithic approach rely on newer frontier models to follow complex instructions across a call. But Varun Singh points out a fatal flaw in long system prompts. LLMs suffer from a severe lost-in-the-middle problem: “LLMs are you know goofy in that sense that always remember the first 4% and the last 4% and everything in between they kind of forget. So if you have a return policy right in the middle of that context, it's very likely it'll hallucinate.”

Singh notes that in multi-turn conversations, uncompacted history destroys reliability. As context grows, models drift: “At 10,000 tokens it would like go off the rails.”

On the other side, infrastructure engineers argue that rigid graph builders create their own traps. Tyler D'Silva explains the boundary: “If you know a little bit more about and you can predict where the conversation's going to go, then you can have a more of a node or graph builder kind of approach. But if you don't know what's going to happen... in outbound, yes there is a dedicated message, but what the caller says back you never know.”

When unexpected human replies derail a graph, the bot gets stuck. To fix this, Steven Diaz advocates decomposing tasks before they break: “Some of the ways that we think about it in Vapi is offloading some of that instruction. So break it down. Sometimes you don't really need that 10, 15 step workflow in the prompt.”

Who's Right (and When They're Wrong)

Both sides are half right. The right architecture depends entirely on caller predictability and latency budgets.

If you are running outbound appointment confirmations or debt collection, the conversation flow is mostly deterministic. Use a workflow graph. Splitting sub-tasks like identity verification, date picking, and payment processing into dedicated nodes prevents the model from skipping compliance steps or mixing up policies.

If you are handling open-ended inbound customer support, a hardcoded graph will frustrate callers who jump between topics. But do not dump your entire 50-page manual into one prompt either. Instead, adopt the approach outlined by Basia Sudol: “We do within our execution engine have the ability to make some of these rules or prompts or guidelines more situational so that we can check if these are relevant right now before we even include them.”

Run a thin orchestrator prompt with broad conversation guidelines. When a user asks about refunds, inject the refund policy dynamically into the immediate context turn, then purge it when the topic shifts.

What to Do With This

Open your voice agent's system prompt today and count the instructions. If your prompt contains more than five sequential steps or exceeds 2,000 tokens, pull out specialized tasks like payment processing or address collection into standalone sub-nodes. Move static policy text into dynamic retrieval blocks that only load when the caller triggers that specific intent.