Key Takeaways

  • Running Google's Veo video model requires four Nvidia H100s for a 10-second clip, exposing how video and long-context inference are bottlenecked by memory capacity and bandwidth rather than raw compute.
  • Positron AI shipped its first customer deployment on FPGAs in 15 months, deploying on Oracle Atlas racks before raising $875 million for custom silicon tape-outs.
  • Instead of waiting in line behind Nvidia and AMD for High Bandwidth Memory (HBM) and TSMC's CoWoS packaging, Positron designs chips around commodity LPDDR5X memory.
  • Hyperscalers filter hardware startups on two operational questions: whether you can secure fab allocation at scale, and whether your power and cooling design fits their existing data center racks.
  • Modern inference runtimes like SGLang and vLLM have compressed model bring-up times from months to hours on new silicon architectures.

The Memory Wall in Video Inference

AI compute is hitting a hard physical ceiling, but it is not the ceiling most people watch. The choke point is no longer floating-point operations per second; it is the physical pipeline moving weights into compute cores.

Video generation brings this problem to light immediately. As Mitesh Agarwal explains: “Video generation models the amount of memory I remember looking at the Google VO model back and you know it needed four H100s to run like a 10-second clip. Was completely memory bound on bandwidth and capacity.”

When a model demands four server-grade GPUs just to hold its parameters and context for a short video, standard hardware economics break down. The answer is not simply buying more expensive GPUs. It requires rethinking memory architecture from the board up.

Skipping the Line for HBM and CoWoS

Every chip designer trying to beat Nvidia with faster HBM faces a brutal manufacturing reality. Nvidia, AMD, and Google command the bulk of TSMC's advanced packaging lines.

“For us our big stories are you know like look HBM and then CoWoS bottleneck you have Nvidia TPUs AMD is ahead of you in that line,” Agarwal notes. “How do you get around that? Well again you know you you say okay we are using commodity memory. Well pro and con no free lunch in silicon land like commodity memory is slow how do you solve that that's where the technical innovation comes in.”

By building around high-volume commodity parts like LPDDR5X, Positron bypasses the packaging backlog entirely. The tradeoff is raw latency, which demands architectural compensations in the chip design itself. But it gives the company a clear path to volume manufacturing while others wait for CoWoS allocation.

What Hyperscalers Actually Ask

When hardware startups pitch cloud providers and large hedge funds, the conversation rarely lingers on architectural theory. Buyers care about deployment physics.

“When you step into the rooms of like not only just Jump Trading or hedge funds or kind of inference service providers but like really the big labs the hyperscalers kind of the two questions that it boils down to is like hey like look guys can you fabricate this in enough quantities,” Agarwal says. “And then the second thing they're asking is can we deploy it you know is is your power source like you know do you need this kind of liquid pool setup.”

At the same time, software risk has changed. Early chip startups spent years writing custom kernels for every new model architecture. Today, flexible runtimes like SGLang and vLLM let teams adapt custom silicon to new model drops in hours rather than quarters.

What to Do With This

Profile your production inference stack this week to identify whether your slowest endpoints are compute-bound or memory-bound. If your batch sizes stall on memory capacity during long-context or multi-modal generation, test an inference runtime like vLLM or SGLang with aggressive KV-cache quantization before ordering higher-tier GPU clusters.