Key Takeaways

  • Renting an NVIDIA H100 GPU on cloud hyperscalers costs between $35,000 and $50,000 per year, compared to an outright purchase price of roughly $30,000.
  • Cloud rental expenses can reach 1.5x the total purchase price in year one alone, while owned silicon carries a 3-year warranty and can run workloads for up to a decade.
  • Rapid chip obsolescence affects frontier model training far more than inference, meaning older hardware can be reassigned to low-latency tasks like text-to-speech without hurting user experience.
  • Startups managing predictable baseline compute save millions by balancing hardware ownership against cloud burst capacity using Speechify's 3-Tier Compute Capacity Allocation Framework.

The Speechify 3-Tier Compute Capacity Allocation Framework

Most AI founders treat cloud compute as a default utility bill. Cliff Weitzman ran the raw hardware numbers and chose a different path. Renting an H100 through GCP costs roughly $5 per hour on spot instances, while AWS or Azure might charge $3.50 per hour. Over a full year of 24/7 uptime, that adds up to between $35,000 and $50,000 per card.

“So, it's 1.5x the cost of owning the hardware to rent the hardware for a year,” Weitzman says. “Now, the hardware is typically warrantied for 3 years to work properly, but it'll keep working after the warranty for I imagine 10 years. So the math just maths where it makes way more sense to buy them.”

To balance cash flow, depreciation risk, and seasonal traffic, Speechify structures its compute capacity across three distinct operational layers:

Tier 1: Owned Hardware (Baseline Load)

Purchase roughly 20% of normalized, predictable baseline GPU compute directly to achieve lowest cost-per-flop economics and dedicate high-end clusters for elite training runs and core inference.

Tier 2: Long-Term Hyperscaler Contracts (Mid-Tier Load)

Commit roughly 25% of compute capacity to multi-year discounted reserved instances with cloud hyperscalers (AWS, GCP, Azure).

Tier 3: Spot Instances (Peak & Seasonal Load)

Fulfill remaining fluctuating and seasonal surge demand (such as back-to-school peaks) via on-demand and spot instances from cloud providers without overcommitting capital.

When This Works (and When It Doesn't)

This framework applies to scale-stage AI companies with steady, predictable monthly training and inference workloads that experience seasonal spikes. When your company processes millions of requests around the clock, paying the cloud margin on every single floating-point operation drains your balance sheet.

When Harry Stebbings challenged the risk of rapid chip depreciation and locked architectures, Weitzman pointed out the distinction between training and inference. For frontier training, speed is everything. Older chips lose their edge in model training, but they remain perfectly suited for production inference where delivering audio in 100 milliseconds is the goal.

This strategy fails for early-stage startups that have not reached product-market fit. If your workloads fluctuate wildly from week to week, or if you lack the cash reserves to purchase silicon upfront, buying hardware introduces fixed overhead that can sink you.

What to Do With This

Audit your cloud spend from the past six months to identify your absolute floor: the minimum number of GPUs your product kept pegged at 100% utilization every single day. If that floor requires more than 10 dedicated GPUs, price out purchasing those units directly through hardware vendors instead of renewing monthly cloud commitments. Model the payback timeline assuming a 36-month hardware life, and relegate cloud instances strictly to handling burst traffic above that base.