Key Takeaways

  • Pure token reselling carries zero switching costs, forcing API providers into an immediate price war.
  • The 90 percent software gross margins common in Silicon Valley a decade ago no longer exist in AI.
  • Modal targets 40 to 60 percent gross margins by bundling inference with sandboxes, post-training, and reinforcement learning.
  • Infrastructure economics require splitting revenue across hardware makers, 25 neo-clouds, model creators, and applications.

Software Margins Are Dead

Silicon Valley spent two decades getting rich on software gross margins north of 90 percent. You wrote code once, hosted it on a single cloud instance, and collected recurring revenue. Erik Bernhardsson says those economics are dead.

Modal now splits revenue across hardware providers, 25 separate neo-clouds, model creators, and end applications. When compute sits at the center of every customer action, cost of goods sold cannot disappear.

As Bernhardsson put it: “I think basically almost no company today has software margins anymore like what you used to see in Silicon Valley ten years ago with 90 percent margins plus. That is basically gone.” (play from 14:07)

Founders building AI products on top of raw compute must face this reality early. If your business model requires 85 percent gross margins to break even, your unit economics will collapse the moment inference usage scales. Modal itself accepts this shift. Bernhardsson targets 40 to 60 percent margins, recognizing that hardware will always take its cut.

Token Reselling Has Zero Switching Costs

The most dangerous position in AI right now is reselling raw model tokens. If your product only offers access to an LLM endpoint, your buyer can switch to a competitor in four minutes by updating one line in an environment file.

Bernhardsson is direct about the trap: “If you're just selling tokens for dollars, there's no switching cost. People can just switch to a different provider. Then it's just a race to the bottom.” (play from 12:41)

Inference itself will grow into a massive sector. Bernhardsson views it as the direct successor to classical cloud computing: “First of all, I think inference is going to be an enormously large market. It's like the new cloud basically. But I don't think inference in itself is maybe like the only thing you need.” (play from 11:55)

When compute acts as a pure commodity, prices fall to marginal cost. Competing on penny discounts per million tokens guarantees your gross margin compresses toward single digits.

The Margin Lives in the Workflow Bundle

Defensible gross margins require an execution environment wrapped around the model rather than bare hosting scripts.

Bernhardsson points to the layers surrounding bare inference: AI sandboxes, reinforcement learning environments, post-training pipelines, and agentic workflows. By aggregating compute across 25 neo-clouds and pairing inference directly with isolated execution sandboxes, Modal builds structural switching costs. A developer might swap an API key for a raw model, but migrating an entire sandbox environment with stateful agent execution and fine-tuning pipelines requires heavy engineering effort.

“One thing that people don't see about the inference market is I actually think in the long term there is so much gross margin opportunity in the space because you can build all these things around it,” (play from 13:30) Bernhardsson noted.

The lesson applies beyond infrastructure providers. Whether you run a developer platform or an AI application, standalone prompts are vulnerable. Defensibility comes from the surrounding state: the code execution, the evaluation loops, and the workflow integration.

What to Do With This

Audit your bill of materials this week. If raw token pass-through represents more than 40 percent of your revenue, introduce a feature that stores proprietary workflow state, such as an evaluation sandbox, an execution container, or a custom post-training loop. Build dependencies around the context, not the model.