Key Takeaways

  • Discrete SRAM chips hold only megabytes of memory per die, forcing systems to link thousands of separate chips simply to store weights for trillion-parameter frontier models.
  • Cerebras packs roughly 100 times more on-chip SRAM onto a single wafer than discrete LPUs, bypassing inter-chip networking penalties.
  • Groq published production benchmarks on a 31 billion parameter model because smaller discrete chips hit a physical scaling wall on larger architectures.
  • High-speed generation at 100 to 200 tokens per second is shifting from a premium tier into the baseline expectation for standard batch processing.

The Memory Trap Behind Fast Inference

SRAM is fast because it sits directly on silicon next to compute cores. It eliminates the memory bandwidth bottlenecks that slow down traditional DRAM. But discrete chips face a hard physical limit: the reticle size of standard semiconductor manufacturing tools caps chip size at roughly 800 square millimeters.

Because discrete chips remain small, each unit holds only tens to hundreds of megabytes of SRAM. Sean Lie acknowledges the industry shift: “In general, I think it's awesome that SRAM designs are becoming more mainstream now.”

Yet the math breaks down quickly as model sizes increase. When running a small 7B or 31B parameter model, a small cluster of discrete LPUs works fine. But frontier models now reach multiple trillions of parameters. Lie points out the structural issue: “to run a frontier level model that let's say a few trillion parameters, you need thousands and thousands of Groq LPUs just to hold the weights.”

When thousands of discrete chips are chained together just to store model weights, the system spends its latency budget routing data across circuit boards, connectors, and cables. The speed of SRAM gets eaten by board-level networking.

The Wafer-Scale Shortcut to Scale

Cerebras took a different architectural bet: keeping the entire silicon wafer intact instead of cutting it into hundreds of small dies.

By keeping the wafer whole, the interconnect stays on silicon. Lie explains the scaling advantage: “in our case, we're running the world's largest models and in some ways kind of simple cuz one of our chips has order 100 times more memory than one of their chips. So we got two orders of magnitude difference in scale kind of for free.”

This physical difference explains why discrete chip makers pick specific benchmarks. Lie noted that hardware engineers at Hot Chips noticed the gap between marketing and benchmark scale: “it's very suspicious that their product that's in full production, they've only shown performance numbers on a non-disaggregated 31 billion parameter model.”

When hardware cannot store weights without spreading them across thousands of discrete parts, running multi-trillion parameter models becomes impractical. The architectural winner for frontier inference will not be the chip with the fastest single-core clock, but the chip that avoids off-die communication entirely.

What to Do With This

Audit your inference vendor contracts this week before locking in multi-year compute commitments. If a hardware provider shows you 500 tokens per second on an 8B or 31B model, ask for their audited latency benchmarks on 70B and 405B models. Calculate how many physical chips their architecture requires to hold your production model weights; if that number exceeds 100 units, expect network overhead to degrade your real-world speed.