Key Takeaways

  • Compute power scaled 1,000,000-fold over the last 25 years, while memory bandwidth improved only 40-fold, creating a 25,000-fold bottleneck.
  • Autoregressive inference forces the processor to reload all model weights from memory for every single token generated.
  • Planar chip architectures separate logic and DRAM into different foundries, forcing signals across narrow lateral chip edges.
  • Fractile stacks DRAM vertically on top of the logic die, routing millions of copper connections straight down across the surface area.

Compute Ran Ahead by a Factor of 25,000

For 25 years, chipmakers chased raw arithmetic execution. Floating-point operations per second scaled by roughly one million times. Yet during that same period, memory bandwidth grew by just 40 times.

Walter Goodwin points directly to the physical limit behind that gap: “bandwidth by contrast is kind of much more intrinsically physics limited. We've scaled it about 40fold in the same time. So you know it's lagged behind the flop scaling about 25,000 times.” (play from 15:27)

The semiconductor industry operated under the assumption that arithmetic speed dictated system throughput. That assumption worked when chips ran classical scientific computing or batch training. It fails during real-time model inference. A modern inference chip spends very little time waiting for arithmetic units to finish an equation. Instead, the chip sits idle, starved of data, waiting for weights to travel from memory.

Lateral Wires Choke Token Generation

Autoregressive token generation strains planar memory setups because of how language models process sequences. To output a single word, the system must inspect the entire parameter set. Goodwin outlines the mechanical penalty: “Every time I go and generate one of those words, I'm loading from that memory over that limited bandwidth all of the weights of my model.” (play from 16:07)

Generating a hundred words requires pulling those billions of parameters through the bus one hundred times in sequence. If that bus cannot move data fast enough, extra compute cores cannot help.

Hardware manufacturing traditions created this boundary. Goodwin explains: “The reason that over the last 30 years we've externalized memory into this this thing called DRAM and put it on different silicon. These are built in completely different foundries.” (play from 17:57) Logic dies require processes tuned for high-speed switching. DRAM dies require processes optimized for dense, cheap capacitor storage. Placing them on separate dies forced data to cross lateral edges. Those edges provide limited physical perimeter space, capping how many copper wires can connect memory to compute.

Stacking Memory Directly Over the Logic Die

SRAM placed directly on the logic die solves the speed problem, but it lacks the density to store tens of billions of parameters. Standard High Bandwidth Memory clusters DRAM stacks beside the processor, but the signals still must travel horizontally across an interposer's edge.

Fractile bypassed lateral wiring by moving upward. Goodwin describes the architectural shift: “what we've been working on at Fractile over the past couple of years is uh essentially stacking that memory on top of the logic.” (play from 19:44)

Going vertical replaces an edge routing problem with an area routing solution. Goodwin highlights the geometric change: “When you put this on top, suddenly you have this whole two-dimensional area where you can have essentially those copper wires are now not going laterally. you have millions of them coming straight down.” (play from 21:05)

By placing the memory directly over the processor, millions of vertical copper connections link the two layers across their entire surface area. Data moves microns instead of millimeters, opening the bandwidth needed to feed inference engines continuously.

What to Do With This

Audit your model serving metrics this week. Measure your cluster's actual core activity during peak inference to see whether your processors are executing math or stalling on memory transfers. If your token generation latency climbs while processor core load stays under 30%, stop spending capital on higher-compute instances and switch your workloads to memory-bandwidth-optimized hardware or aggressive weight quantization.