Key Takeaways

  • Luminal raised $25 million to build compilers that run models on alternative chips like Amazon Trainium and Positron Atlas.
  • Software, not silicon manufacturing, protects Nvidia's market position, because dozens of hardware teams can tape out chips but fail on compiler stacks.
  • Writing a compiler backend takes less time than taping out an ASIC, allowing software teams to enter late in the hardware design cycle.
  • In commercial inference handling 100 trillion tokens a month, buyers choose cheaper throughput over faster raw speed almost every single time.
  • Amazon Trainium delivers better price-performance than Nvidia Blackwell Ultra on production inference workloads once latency hits acceptable minimum thresholds.

Silicon Is Commoditizing Faster Than Compilers

Hardware engineers know how to design fast chips. The difficulty is making those chips actually run code written in standard machine learning frameworks. Nvidia has held its lead not because TSMC cannot make chips for other companies, but because CUDA works out of the box while competitor software stacks fail.

Luminal CEO Joe Fioti founded his company to exploit this gap, closing a $25 million funding round to build what he calls a software-defined accelerator company. As Fioti explains: “Basically all AI accelerators have a hardware component and a software component. And what we realized is there is actually a lot of accelerator companies that can make the hardware component and only one that can kind of make the software component.” (play from 1:56:24)

Building silicon takes years of physical engineering, cleanroom scheduling, and tens of millions of dollars in foundry costs. Yet software moves faster. Luminal steps into the chip production cycle after the physical architecture is already locked in. “Really what we are concerned with is it actually is faster to bring up a compiler backend than it is to tape a chip out,” (play from 1:59:47) Fioti said. “So we actually come in pretty late in the process of building a chip. They generally come with a completed design to us. We kind of figure out the lowest level software primitives and then go on up from there.” (play from 1:59:53) By treating silicon as a fixed target and writing the translation layer, Luminal can adapt models to new hardware without waiting for hardware redesigns.

Buyers Pick Cheaper Over Faster Every Time

The AI industry often obsesses over peak latency benchmark records. Hardware makers boast about shaving milliseconds off single-token response times. But commercial customers running enterprise workloads have different priorities. They look at the monthly cloud bill.

Luminal processes roughly 100 trillion inference tokens a month across its platform. At that scale, speed matters only until it clears an acceptable user experience threshold. Once a model responds fast enough for a human reader or an API worker, extra speed offers diminishing returns.

“It all comes down to cost,” (play from 1:58:31) Fioti noted. “Speed is good, but speed beyond a threshold is not super useful. People almost 10 times out of 10 when given faster or cheaper will pick cheaper. And so we are always very focused on throughput, and we really like accelerators that have a big throughput focus.” (play from 1:58:32)

This dynamic makes non-Nvidia hardware commercially viable today. Luminal runs heavy customer traffic on Amazon Trainium and Positron Atlas chips. When tested against top-tier hardware, the alternative silicon wins on economic efficiency. “We get, even against like Blackwell Ultra, pretty phenomenal price performance on Trainium,” (play from 1:58:57) Fioti said. Once software compilers eliminate the technical barrier to switching, the economics force large-scale buyers away from Nvidia's high-margin products.

What to Do With This

Audit your inference spend this week. Break down your token volume by workload type, separating real-time customer-facing queries from asynchronous batch jobs like evaluations, data ingestion, and background agents. For every asynchronous job where response latency under two seconds does not change business outcomes, run a benchmark test on Amazon Trainium or an alternative inference provider to compare raw dollar cost per million tokens against your current Nvidia cluster.