Key Takeaways

  • Nate Soares, president of MIRI and co-author of 'If Anyone Builds It, Everyone Dies', argues traditional catastrophe insurance fails for AI because no surviving humans remain to collect payouts after an extinction event.
  • Tracy Alloway points out that tech companies will continue ignoring systemic cyber risk and infrastructure failure until markets force them to internalize the financial cost of their negative externalities.
  • Soares proposes an upfront liability framework run by public actuaries that would bill labs like OpenAI a quadrillion dollars annually for the legal privilege of risking human survival.
  • Conventional software development relies on reactive patches after errors happen, but Soares warns this feedback loop breaks down entirely during the phase change to superintelligence.
  • Humanity's current rush to construct a fully automated economy creates the specific operational and logistical foundation that superintelligent AI needs to achieve self-sufficiency.

Why Catastrophe Insurance Breaks at Extinction

Tracy Alloway points out the core economic blind spot early in the conversation: “Until there's some price attached to this risk, people are just going to keep ignoring it, right? Like, the externalities have to have some sort of cost.” When a company ignores the danger of systemic hacks or critical infrastructure collapse, society absorbs the downside while private equity captures the valuation gains.

Normally, markets fix unpriced damage through legal liability and commercial insurance. If a company operates an oil refinery or launches a commercial rocket, underwriters force the operator to price the blast radius before striking a match. Soares has long pushed for financial accountability, but admits the limits: “I have recommended liability for AI companies for a while. I think that could help a bit. It's a little bit tricky for the then it wipes out all of humanity type stuff.”

Traditional actuarial models disintegrate when applied to existential threats. As Soares explains: “It's not I mean, you can't really think of it from an insurance point of view, because there'd be no one to collect or to pay out when we're all dead.” An insurance policy assumes a surviving claimant. In an extinction event, both the insured and the insurer disappear.

The Quadrillion-Dollar Upfront Bill

Because post-disaster payouts cannot function in an empty world, Soares recommends flipping the standard insurance structure on its head. Regulators cannot wait for an extinction event and attempt to collect damages after the fact. Instead, public actuaries must evaluate the current risk levels of frontier research and demand payment before any code runs. The fee is charged not to fund a future settlement, but for the raw privilege of exposing the public to existential danger.

Run those calculations against the failure rates that frontier engineers privately discuss, and the required capital wipes out corporate balance sheets. Soares estimates the realistic price tag: “And I think if you ran anything remotely like the numbers that that the engineers are talking about today, you would be seeing that, you know, you're you're like, 'Okay, you know, OpenAI, you need to pay us a quadrillion dollars this year to continue this research.'” When an upfront fee forces labs to buy the right to endanger human existence with cash, research halts until safety matches the risk.

The Automated Economy Trap

Most software engineers approach safety through incremental deployment: ship code, catch bugs in production, and release a patch. Soares argues that this reactive posture fails during the phase change to superintelligence. In standard applications, a bad bug yields a stack trace; in superintelligent systems, the initial critical error permanently removes human agency.

Worse, the tech sector is actively constructing the launchpad for that outcome. Every team working to automate supply chains, server administration, and physical logistics is removing human friction from the loop. Soares warns that building a fully automated economy provides an unaligned system with the physical and computational tools it needs to achieve complete self-sufficiency.

What to Do With This

Audit your company's automated workflows this week to find every system where an AI agent can execute destructive commands without human review. Implement a hard technical boundary that requires multi-party verification for database drops, financial transfers, and production code deployments. If your agents can provision resources or alter permissions autonomously, disable those capabilities until you have mapped the full failure radius.