Key Takeaways

  • Physical transistor scaling has stalled: Dennard scaling no longer yields automatic energy efficiency gains with each chip shrink.
  • Most efficiency gains over recent generations came from precision reduction, shifting from 32-bit floating point calculations down to 16-bit, 8-bit, and 4-bit formats.
  • Rao frames computing as a direct economic conversion: “The conversion of energy to intelligence is hardware... I'm going to be selling the capability to convert energy to intelligence.”
  • Pure software SaaS businesses face structural threats because energy constraints force models and physical silicon to merge into integrated systems.
  • Rao evaluates compute performance and technical advantage through the Three-Level Computing Efficiency Hierarchy.

The Three-Level Computing Efficiency Hierarchy

To understand why traditional computing roadmaps are broken, Rao separates how we build computers into three distinct layers:

Level 1: The Physical Layer

Making devices physically smaller and more efficient at moving a bit between states (Dennard scaling and Moore's law). Innovations here have largely stopped as transistors have reached physical limits.

Level 2: The Computing Architecture Layer

Gaining efficiency through smaller data types and precision reduction: transitioning calculations from 32-bit to 16-bit, 8-bit, and 4-bit representations to compute with and move fewer bits.

Level 3: The Model-Hardware Co-Design Layer

Enforcing spatial locality and reducing data movement by co-designing the model with the physical hardware (for example, matching sparsity in neural networks to physical memory structures rather than relying on off-chip RAM).

When This Works (and When It Doesn't)

This framework works when you are deciding where to invest engineering payroll across an infrastructure stack. For decades, software teams treated the underlying silicon as a magic black box that doubled in performance every eighteen months for free. You could write inefficient code, deploy bloated weights, and let Nvidia or Intel solve your throughput problems. That era is over. Level 1 is tapped out. The gains from Level 2 are narrowing as quantization approaches single-bit representations where mathematical degradation ruins model accuracy.

The edge case where this framework breaks down is small-scale experimentation. If you are building a prototype with ten daily active users, spending engineering hours co-designing memory pipelines or customizing model layers to match on-chip SRAM is a waste of capital. Commodity cloud instances and standard off-the-shelf APIs are fine for testing distribution. Co-design only matters once compute volume scales to the point where power bills and latency constraints threaten the existence of your gross margins.

What to Do With This

If you run an AI engineering team today, stop treating the model as fixed. Rao notes, "we don't treat the model as sacrosanct... everything we do has to give us an energy advantage." Run an audit of your inference stack this week using the three levels:

1. At Level 1, accept that future chip generations will not rescue your unit economics on their own. Strip out any assumptions in your 2026 financial model that rely on 50% physical hardware cost drops.

2. At Level 2, benchmark your production workloads across precision formats. If your production endpoints still run on FP16 or FP32 weights, test INT8 and FP4 quantization on your primary eval sets tomorrow morning. Cutting precision halves memory bandwidth pressure instantly.

3. At Level 3, examine where off-chip memory bandwidth chokes your latency. Restructure your model architectures to keep working memory inside local chip cache, pruning activation states to match physical memory layouts rather than blindly scaling parameter counts.