Key Takeaways

  • To handle massive 200,000-token LLM queries cost-effectively, Baseten first checks if parts of the input have been seen before, using cache-aware routing to skip expensive re-computations.
  • They split the LLM inference pipeline into two distinct stages—prefill and decode—running them on separate sets of GPUs to optimize resource utilization and throughput for mixed workloads.
  • Speculative decoding accelerates token generation by predicting future tokens. This method becomes particularly effective when an initial encoding assumption aligns with the model's behavior.
  • These aren't just minor tweaks; they're architectural redesigns, treating LLM inference as a complex distributed system problem to unlock performance and reduce costs at scale.

The Method

Founders chasing performance on long-context large language models (LLMs) often hit a wall: it's incredibly slow and expensive. Philip Kiely from Baseten laid out their blueprint for tackling this, specifically for massive 200,000-token queries. His team focuses on three core technical moves to make long-context LLM inference not just possible, but efficient.

First, they attack the most wasteful part: redundant computation. “With a long query specifically, the first thing that I'm going to ask is, have you sent me this query before, or at least part of it?” Kiely explained. This isn't just a simple cache; it's cache-aware routing. They direct queries to specific GPU instances that already hold the input's key-value (KV) cache. This means skipping the entire, costly prefill operation for those tokens. As Kiely put it, “we want to send this one to something with number one available prefill workers, and number two, ideally some cached input already there so that we can skip prefill on at least part of this 200,000 tokens.” It's about intelligently matching workloads to resources with existing state.

Next, Baseten saw that the two main phases of LLM inference—prefill (processing the input prompt) and decode (generating new tokens)—have wildly different computational profiles. So, they disaggregated prefill and decode. Kiely noted, “we've at least on certain models disaggregated prefill and decode. You're going to have one set of GPUs that's solely going to process the input query, that KV cache, and get you your first token. And then that's going to be passed over to a separate set of GPUs which is going to run decode.” This architectural split lets them use GPUs more effectively. Prefill, often bursty and compute-heavy, can run on dedicated resources, while decode, which is more sequential, runs on others. This prevents bottlenecks and ensures better hardware utilization, especially under varying loads.

Finally, they layer in speculative decoding to speed up token generation itself. This technique involves using a smaller, faster "draft" model to predict a sequence of tokens. The larger, more accurate model then verifies these predictions in parallel. If the draft is good, many tokens can be accepted at once, dramatically accelerating output. Kiely pointed out that "a speculative model which assumes you do encoding is going to have a high draft token acceptance rates." This is a smart bet on common encoding patterns to get faster results.

Where This Breaks Down

These advanced techniques offer serious gains, but they aren't a free lunch. Implementing cache-aware routing, prefill/decode disaggregation, and speculative decoding demands significant upfront engineering effort and a sophisticated infrastructure. For smaller teams or products with low inference volume, the return on investment might not justify the complexity. For instance, cache-aware routing only helps if you have a reasonable cache hit rate; if every query is unique, you gain nothing. Disaggregating prefill and decode adds overhead in scheduling and managing multiple GPU clusters, which can become a headache if not done carefully. Speculative decoding's effectiveness also hinges on the accuracy of the draft model. If it's constantly wrong, the main model has to re-verify each token individually, negating much of the benefit.

What to Do With This

If you're a founder building any compute-intensive service, not just LLMs, step back and look at your pipeline. Identify the distinct phases in your most expensive operations that have different resource needs, just like Baseten did with prefill and decode. Can you disaggregate these stages onto different hardware or processes for better utilization? Tomorrow, audit your core business logic: are you repeatedly computing the same inputs? Implement intelligent caching and routing to avoid redundant work, even if it's not a full KV cache. If you're building an LLM product, challenge your engineering team: specifically ask how they handle long context. Don't settle for vague "optimizations." Probe them on cache-aware routing, prefill/decode disaggregation, and their approach to speculative decoding. Their answers will tell you how serious they are about managing your compute costs at scale.