Key Takeaways

  • Frontier data centers scaling from hundreds of megawatts to gigawatts must treat the entire facility as a single disaggregated computer rather than a uniform grid of identical GPUs.
  • Inference is fragmenting into distinct sub-workloads: prefill, decode, KV cache loading, and Mixture-of-Experts (MoE) routing, each demanding different memory and compute profiles.
  • Output speeds of 100 to 200 tokens per second are rapidly turning into the new batch processing baseline, while wafer-scale roadmaps like Cerebras CS-5 push toward 10,000 tokens per second.
  • Current frontier AI models remain heavily over-optimized for single Nvidia GPU constraints. Cerebras records a 14x speedup running unmodified models, pointing to even larger performance wins through co-design.

The End of the Identical GPU Cluster

Every major AI lab built its early lead by buying thousands of identical Nvidia GPUs, networking them together, and treating the cluster as a uniform grid. Sean Lie points out that this monolithic setup falls apart as data centers scale toward multi-gigawatt footprints.

When a single facility draws hundreds of megawatts, running every stage of model execution on the exact same chip creates massive inefficiency. Prompt processing (prefill) requires high compute density, while token generation (decode) demands raw memory bandwidth and ultra-low latency. Mixing these workloads across identical chips forces hardware compromises on both ends.

As Lie explained, “inference is no longer just like one workload. There's many many kind of sub workloads within it. And, you know, you really want to use the right you know, tool for the for the problem.” Instead of deploying thousands of identical boxes, operators must build disaggregated environments where dedicated silicon handles prefill, SRAM-heavy wafer-scale engines run decode, and specialized memory nodes manage the KV cache.

At gigawatt scale, the operational savings easily offset the software complexity of routing between specialized chips. Lie noted, “at the scale we're all talking about, the hundreds of megawatts to gigawatts to multi-gigawatt scale, it easily pays off. And then at that point, it's as if you're like thinking about the entire data center as if it's like one computer.”

The Trap of Nvidia-Native Architecture

Almost every modern model architecture, from attention head counts to parameter sizes, was chosen because it fit the memory hierarchy and tensor core shapes of Nvidia GPUs. AI researchers spent years designing architectures around the exact limits of high-bandwidth memory (HBM) and DRAM on standard PCIe and SXM boards.

This creates an artificial ceiling for the industry. Lie emphasized that the biggest opportunity in machine learning hardware is breaking away from assumptions inherited from Nvidia silicon: “what I think is um the most untapped opportunity right now, frankly, for Cerebras, but frankly, for the entire non-Nvidia environment... is that, you know, we're all running models that were designed for Nvidia GPUs.”

Even when running weights configured strictly for standard GPUs, Cerebras showed that its wafer-scale architecture delivered dramatic improvements. Lie pointed out, “here we were showing that you're running the model like 14 times faster already, but it was a model that was designed for a completely different architecture. And so, if you start to then open up the possibility of adjusting that model architecture, even slightly, you can get massive gains.”

When teams adjust model architectures to match alternative memory layouts, such as massive on-chip SRAM instead of off-chip DRAM, latency drops by orders of magnitude. The next generation of performance breakthroughs will not come solely from shrinking transistors, but from co-designing model topologies alongside the silicon that runs them.

What to Do With This

Profile your production inference traffic this week and split your telemetry into prefill time-to-first-token versus decode inter-token latency. If you run large prompts with interactive agentic loops, spin up a test endpoint on a non-GPU, high-bandwidth engine like Cerebras to serve the decode phase separately from your prefill cluster. Adjust your model's attention head dimensions during your next fine-tuning cycle to evaluate throughput gains when freed from standard GPU tensor core constraints.