Key Takeaways

  • Kush Bavaria compares frontier AI training to speculative biotech R&D, where founders burn cash without knowing if the model will produce a viable outcome.
  • Inference spending reflects verified usage because buyers only pay when someone consumes actual tokens.
  • Depreciated hardware will not sit idle; older chips rotate out of frontier clusters to handle lower-scale inference and smaller open-source models.
  • The compute market will split into a single horizontal layer for baseline tasks and dozens of specialized engines tailored for video, chemistry, and biological computing.
  • Compute platforms face pressure to reject easy short-term revenue from buyers like Modal, Fireworks, and Fal to protect their long-term supply positions.

Training Is Pure Speculation While Inference Tracks Actual Usage

Venture capital and tech giants poured tens of billions into training runs over the last two years. Bavaria points out that this capital expenditure resembles drug discovery far more than traditional software infrastructure: “Training is much more speculative, right? It is like kind of like biotech where you spend all this money developing this drug and you don't actually know if it works out or doesn't work out.” (play from 42:36)

When a lab funds a frontier model, every dollar is an unhedged bet on capability jumps. The compute cluster gets burned on weights that might underperform, become obsolete overnight, or yield zero commercial demand.

Inference operates on opposite economics. “The good thing about the inference demand is it's not speculative. It is like real,” (play from 42:23) Bavaria explains. “The tokens have to be used by someone.” (play from 42:27)

Because inference requires an active end user triggering an API call, it creates a true consumption floor. A startup cannot run inference compute on hope. If users stop pinging the endpoint, token consumption halts immediately. This difference changes how infrastructure debt and capacity must be underwritten across the market.

Older Chips Will Power Specialized Inference Verticals

Hardware cycles create an assumption that older GPUs become worthless electronic scrap once next-generation architectures ship. Bavaria argues that chip depreciation curves will stabilize because older processors slide into distinct operational tiers: “I also believe that like the trip depreciation like the older chips will have a use besides being like frontier model training. They'll be used for lower scale inference, smaller model inference.” (play from 18:09)

Frontier labs will always require top-tier clusters with maximum inter-chip bandwidth to train state-of-the-art models. But as open-source models shrink in parameter size while matching earlier performance benchmarks, older silicon runs day-to-day queries cheaply.

This dynamic shapes how inference capacity gets organized. Instead of one or two consolidated winners capturing every query, the market will break into vertical silos: “I think there will be like a horizontal provider that is like very good at like providing like base level inference. Um, that will be the everyday person's like inference company, but then there will be like a hundred different specialized inference companies that exist.” (play from 19:46)

A general chatbot query does not demand the same underlying stack as real-time video generation, molecular modeling, or biological sequence analysis. Workloads requiring raw token speed or high-throughput scientific calculations require dedicated, purpose-built engines.

For infrastructure providers and neo-clouds, capturing that value requires discipline. Bavaria notes that operators must often resist selling out all their capacity to intermediary platforms: “You have to say no to a modal. You have to say no to a fireworks. You have to say no to a foul which is willing to pay you gazillions of dollars in revenue and profit for your underlying business to like kind of look in the long term.” (play from 40:35)

What to Do With This

Map your inference spend across your engineering stack this week. Split your workloads between frontier queries that require expensive hardware and utility tasks that run on smaller models. Migrate the smaller tasks to older, depreciated chip clusters or specialized secondary providers to cut your token costs before signing long-term capacity commitments.