Key Takeaways

  • Model training is shifting into a fixed upfront cost, making inference and test-time compute the primary operational expense of frontier intelligence.
  • Tackling heavy math and scientific challenges, such as the Navier-Stokes Millennium Prize problem, demands swarms of 10,000 agents generating tokens across 80 hours of continuous execution.
  • Existing hardware forces an ugly compromise: SRAM chips generate tokens fast but run out of memory, while HBM GPUs hold context but choke on sequential token latency.
  • Millions of sequential tokens create multi-day wait times because modern data center silicon was never architected for autonomous, persistent reasoning loops.

Test-Time Compute Is the New Scaling Frontier

For years, progress meant larger training clusters. Teams threw thousands of processors at web-scraped corpora for three months, published a checkpoint, and served queries. Walter Goodwin argues that dynamic is dead. As Goodwin puts it, “training becomes a fixed cost and the deployment, inference, is just the use of these models to do useful things. It's the only part that matters.” (play from 42:51)

Once a frontier model exists, you cannot simply retrain it every week to answer harder questions. You extend its thinking time at runtime instead. Goodwin points out that when researchers have access to the world's best model, test-time compute is “the knob that we have to turn to get greater capability.” (play from 46:07) Instead of returning a fast guess in two seconds, the system ponders, explores branches, critiques itself, and writes intermediate tools. The work shifts from a single forward pass to millions of generated tokens.

The Navier-Stokes Test and the 80-Hour Wall

Real problems do not look like chatbot queries. They look like open-ended research. Goodwin highlights what it actually takes to address hardest-class problems: “If you look at the examples of solving a Millennium Prize problem, that Navier-Stokes solution was 80 hours of execution from 10,000 agents in a swarm.” (play from 44:27)

Picture that workload: 10,000 parallel workers, each pursuing its own line of inquiry, coordinating with peers, and running unbroken for three and a half days. “Where I've both got 10,000 parallel agents all generating their own chains of thought and then they're running for 80 hours,” (play from 44:51) Goodwin says, the hardware requirements blow up. You need vast memory pools to retain the state of thousands of agents, yet you also need blistering single-token speed because each chain of thought is strictly sequential. You cannot parallelize step ten until step nine finishes.

Why Modern Silicon Traps the Builder

This is where existing silicon stalls. Hardware designers have given software teams two bad options.

Option one is pure SRAM. SRAM sits right on the logic die. It feeds tokens at mind-boggling speed because memory access latency is practically zero. The catch: it physically caps out at tiny capacities, typically under one gigabyte per chip. You cannot park long agent memories, system prompts, tool schemas, and swarm coordination logs in SRAM.

Option two is High Bandwidth Memory (HBM) stacked around a GPU. HBM gives you the memory capacity to host big models and large context windows. But reading those weights for one sequential token requires huge amounts of energy to shuffle data across silicon interposers. When an agent runs through millions of sequential reasoning steps, single-token latency dominates. As Goodwin notes: “the many many millions of sequential token generation means you're waiting days and days and days to get one of these responses. And it's a space which somewhat tantalizingly hasn't been cracked at all by any of the silicon any of the chips that we have today.” (play from 24:51)

What to Do With This

Audit your multi-agent architecture this week for sequential token bottlenecks before scaling your swarm size. Open your agent logs and calculate the ratio of sequential generation steps to parallel tool calls across your longest tasks. If a task requires more than 50,000 sequential tokens, price out the latency and compute bill on standard HBM infrastructure, then set an aggressive token budget or prune context early to avoid multi-day runtime traps.