10+ hours of podcasts, in 5 minutes.
5 quotes from 1 episode on Latent Space, each with a timestamped link to the source.
Akshat Bubna explains that AI infrastructure teams now build tools for agents and optimize inference speed. Batching the verification of tokens predicted by a small draft model yields a 2-4x speedup in compute efficiency.
Speculative decoding relies on a small draft model predicting tokens before a large model verifies the output.
“…speculative decoding is you have a smaller model, called a draft model, predict tokens ahead of the bigger model and then you have the bigger model verify all of this.”
From Modal's Dlash: 2-4x LLM Inference Speed, No Quality Loss
Dashboard quality dictates how well teams observe and monitor AI systems.
“…one thing we actually still see is really important is observability. How good is your dashboard?”
From Modal CTO: Agent Experience Is the New Developer Experience
Human workers evaluate system performance and make judgment calls on the collected data.
“You still need humans to go interpret what's going on and make judgment calls and whatnot…”
From Modal CTO: Agent Experience Is the New Developer Experience
A smaller draft model predicts tokens ahead of time. Batching the verification step for a large model improves compute efficiency and generates a 2-4x speedup.
“…if you can batch the verification of the draft model then you're much more efficient using compute and it's faster.”
From Modal's Dlash: 2-4x LLM Inference Speed, No Quality Loss
Software development kits now target AI agents as the primary users. Akshat Bubna shifted a team focus to agent experience to make infrastructure accessible through simple code decorators.
“We've actually changed our SDK team to think about agent experience instead of developer experience…”
From Modal CTO: Agent Experience Is the New Developer Experience
How we attribute quotes. Every quote was matched against the episode transcript, so the words and the timestamp are real (we trim filler words like "um", nothing else). The name comes from our written summary of the episode. YouTube gives us no voice-by-voice transcript, so open the timestamp to hear who is talking. See a wrong name? Tell us and we fix or remove it.
10+ hours of podcasts, distilled into one 5-minute read. Free, every Sunday morning.
One email a week. Unsubscribe with one click.