Key Takeaways

  • AI inference splits into two buying patterns: casual buyers taking spot tokens on price, and mature enterprises paying for reserved physical hardware with strict service level agreements.
  • Spot token generation behaves like an oil or electricity market, where high margins today will face relentless downward pricing pressure as supply catches up.
  • Capital availability and physical chip supply, not electrical grid capacity, remain the immediate bottlenecks choking model deployments today.
  • The rise of autonomous background agents redirects infrastructure demand away from raw GPU hours toward CPU cycles, persistent storage, and dynamic code execution sandboxes.

On-Demand Tokens Are Commodities, Reserved Capacity Is Not

Every founder building on large language models assumes they are buying software. Mensch sees it differently: inference is a utility trading on an open exchange. “The inference market behaves very much like a commodity market which is pretty high margin today,” (play from 31:44) Mensch observes. If you control supply and maintain acceptable response speeds for specific model classes, uncommitted customers will buy tokens on demand at whatever spot price you quote. They care about latency, uptime, and cost per million tokens. The moment another provider serves the same open weights ten percent cheaper with equivalent latency, those buyers switch endpoints without hesitation.

That dynamic creates a trap for infrastructure startups that rely solely on self-serve developer traffic. When consumer-grade token generation is a pure commodity, high margins cannot last. Real enterprise value lives on the other side of the market. “For customers that are sufficiently mature, they need to have guaranteed quality of services,” (play from 32:06) Mensch notes. “They need to have service level agreements and there they need to have reserved physical capacity.” (play from 32:10)

Large enterprises cannot run core operations on shared API pools that degrade during peak hours. When a financial institution or healthcare provider puts models into production, they do not buy shared tokens. They lease physical clusters behind closed doors. They demand contractual throughput floors, dedicated hardware allocation, and data isolation. For model builders, surviving the commodity crunch means moving up the stack from selling metered tokens to managing reserved fleet capacity.

Autonomous Agents Shift the Cost from GPUs to Sandboxes

Most AI budgets in 2023 and 2024 were planned around a single line item: GPU compute for autoregressive decoding. That budget model breaks down the moment systems stop generating conversational text and start acting as autonomous operators. An agent does not just emit sentences. It browses web pages, spins up isolated interpreters, tests generated code, verifies results against external databases, and retries failed steps in loops that run for hours.

That workflow reshapes the underlying data center footprint. Mensch points out that running autonomous workers requires “what we call ancillary services, like running sandboxes, scaling up and down sandboxes very easily when you're running reinforcement learning algorithms for instance, or when you're deploying many agents on the same tasks.” (play from 29:41) GPUs handle model generation, but they sit idle while external environments execute code. “When you start to have managed agents that are running on the background doing a lot of tasks that are CPU intensive, you need more CPUs, you need more sandboxes, and you need more storage as well.” (play from 30:26)

If you build agentic products, your technical margin risk is no longer just the token cost from your model provider. It is the cost of managing thousands of concurrent, short-lived virtual machines, orchestrating execution state, and paying for high-throughput storage read-write cycles. The teams that win enterprise contracts will not be the ones with the cleverest prompt chains; they will be the ones who manage runtime infrastructure without letting sandbox overhead eat their gross margins.

What to Do With This

Audit your infrastructure bill from last month and separate raw model token spend from ancillary services such as vector storage, VM runtimes, and worker queues. If you run background agents, calculate your exact cost per completed task rather than your cost per thousand tokens. Tomorrow morning, negotiate with your infrastructure vendor to shift your core production workloads from spot API pricing to reserved instance capacity with contractual latency guarantees.