Why AI Judges Pick Bad Software: Lessons from Kill My SaaS
When swyx ran an automated Claude eval across 72 software submissions, the winner ranked 19th. Here is why AI judges miss product taste.
How to kill your SaaS: what we learned replacing a $43K/yr tool with AI agents
With swyx, Kelsey, Marko Kraemer, Yazin, Valentin De Matos · Sunday, October 18, 2026
After being quoted $43,000 annually for speaker management software, the AI Engineer team launched the 'Kill My SaaS' competition challenging builders to recreate the software using autonomous coding agents. Across 72 submissions averaging 91,000 lines of code, the hosts discovered stark divergences between automated AI evaluations and human operational taste, documented distinct multi-agent orchestration architectures, and weighed the real-world trade-offs of owning custom software versus renting off-the-shelf SaaS.
Key takeaways
- An automated Claude eval running 20 user scenarios failed to predict human satisfaction: six of its top 10 apps were rated weak by real operators. Read more →
- Commercial software vendors quoted the AI Engineer team $43,000 per year to handle speaker management across four events. Read more →
- When enterprise software vendors quoted $43,000 annually for speaker management software, the AI Engineer team ran the Kill My SaaS competition, generating 72 entries averaging 91,000 lines of code. Read more →
- Faced with a $43,000 annual SaaS quote for speaker management software, the AI Engineer team ran the "Kill My SaaS" contest, drawing 72 submissions averaging 91,000 lines of code. Read more →
4 articles from this episode
Why Killing Your $43,000 SaaS Bill Creates a 2 A.M. Problem
Building custom internal software with AI agents costs $11,000, but owning code means trading bad vendor support for 2 a.m. bug fixes.
How Builders Cloned a $43,000 SaaS With Autonomous Agents
When SaaS vendors quoted $43,000, AI Engineer builders cloned 90k lines of code. Prompting came last after deep specs and trace testing.
Why Top AI Builders Never Let Agents Approve Their Own Code
swyx analyzed 66 codebases from the Kill My SaaS contest to reveal three multi-agent patterns uniting Claude Code and OpenAI Codex.
More Latent Space episodes
- Robots That Predict the Future: Dyna-2 & Physical AI — Jason Ma, Dyna RoboticsOct 18 · 4 articles
- The Bitter Lesson of Biology: Why Scaling AI Isn't Enough — Pushmeet K, DeepMind & Sal C, BiohubOct 11 · 4 articles
- AI Scientists Are Here: Autonomous Labs & Synthesis Superintelligence — Periodic LabsOct 11 · 5 articles
- The Future of Claude Code: Mods, Mutable Software, & Multiplayer Agents — Thariq Shihipar, AnthropicOct 4 · 4 articles