Key Takeaways

  • Modal has open-sourced Dlash, a block-based speculative decoding technique designed to accelerate LLM inference without compromising output quality.
  • Dlash achieves a 2-4x speedup by employing a smaller "draft" model to predict tokens ahead, allowing the larger model to verify predictions in batches, efficiently using compute.
  • This method maintains full quality because the larger, primary model always verifies and accepts or rejects the draft model's predictions.
  • Modal democratizes this frontier-level performance through "auto endpoints," allowing builders to deploy optimized models with Dlash and advanced autoscaling directly from a UI or CLI.
  • These auto endpoints offer significant operational advantages like true scaling to zero and burstiness, which matter more to founders than simply finding raw GPU capacity.

The Method: How Modal Accelerates LLM Inference (Dlash)

Running large language models in production is often a battle against latency and cost. Every token generated by a powerful LLM requires significant compute, and traditional methods often leave GPUs waiting on memory bandwidth. Akshat Bubna, CTO of Modal, recently detailed how his team is tackling this bottleneck head-on with an innovation called Dlash.

Modal's core insight? Don't just run the big model. Instead, run a smaller, faster model alongside it to guess what comes next. Bubna explains, “speculative decoding is you have a smaller model, called a draft model, predict tokens ahead of the bigger model and then you have the bigger model verify all of this.” The magic happens in the verification. If you're verifying one token at a time, your system is often bottlenecked by memory. But if the draft model can predict a string of tokens, the larger model can verify them as a batch. This batching dramatically improves compute efficiency.

This technique, specifically Modal's block-based Dlash, delivers a significant performance boost. Bubna notes, “if you can batch the verification of the draft model then you're much more efficient using compute and it's faster.” The result? A 2-4x multiplicative speedup in inference, according to Bubna, without any compromise on output quality. As he puts it, laughingly, “there's no drop in quality performance because you're always you're never accepting a token that better or same.” The big model always has the final say, ensuring accuracy remains paramount.

To make this frontier technology accessible, Modal created "auto endpoints." This means you don't need to dive deep into custom code to get these optimizations. Bubna says, "sometimes people actually don't want to touch the code and they want to get started with an endpoint that works and has all the great performance and scalability that Modal has." These auto endpoints, deployable from Modal's UI or CLI, come with Dlash and intelligent autoscaling pre-baked. This gives founders not just speed, but also "true scaling to zero" and "true burstiness," crucial for managing unpredictable AI workloads.

Where This Breaks Down

While Dlash offers compelling speedups, its effectiveness hinges on the draft model's quality. If the smaller model consistently guesses poorly, the acceptance rate for its predicted tokens drops, reducing the overall speed gain. The 2-4x boost is achievable when the draft model is well-tuned to the task and the larger model. For highly niche or complex language generation tasks where a generic draft model might struggle, the performance uplift could be less dramatic.

Furthermore, while Modal's auto endpoints simplify deployment, they are an opinionated solution. For builders who require extreme, low-level control over every aspect of their inference stack for highly experimental or proprietary models, directly integrating Dlash into their own custom infrastructure might still be preferred. Modal offers transparency and code-level access, but the "auto" path is designed for speed and convenience over bespoke engineering from scratch.

What to Do With This

If you're running LLM inference and struggling with latency or cost, take an hour this week to explore Modal's auto endpoints. Identify a current LLM application in your product that could benefit from a 2-4x speedup – perhaps an agent workflow, content generation, or summarization. Deploy a test endpoint with one of Modal's optimized models and compare its performance and cost against your existing setup. Understanding how a platform like Modal democratizes frontier inference will either improve your current stack or highlight the path for your own team to achieve similar gains.