Key Takeaways

  • An automated Claude eval running 20 user scenarios failed to predict human satisfaction: six of its top 10 apps were rated weak by real operators.
  • The eventual competition winner ranked 19th on the automated leaderboard, exposing a sharp rift between test coverage and product taste.
  • Builders hill-climbed the synthetic benchmark so fast that the top seven submissions landed within 2.3 points of each other, creating false saturation.
  • Teams like Open Stash found their best UX choices were invisible to LLM graders, which only score explicit, literal checklist specs.

The Failure of Hill-Climbing Against Synthetic Benchmarks

When vendors quoted the AI Engineer team $43,000 a year for speaker management software, swyx ran a contest called "Kill My SaaS." The goal was simple: use autonomous coding agents to build a replacement. Across 72 submissions averaging 91,000 lines of code, the organizers set up an automated grading system using Claude. The AI judge simulated 20 user scenarios to give developers quick feedback.

It worked well for basic progress. As swyx noted: “It was not meant to pick the winner. It's just a sanity check for people to get their applications up and running. Basically, an RL environment without the RL part... a sort of environment for people to hill climb.”

Builders climbed that hill instantly. The top seven submissions finished within 2.3 points of each other. On paper, the contest was a dead heat of high-scoring software. In practice, the synthetic benchmark was scoring compliance, not usability.

The Winner Was Ranked 19th

When human operators sat down to click through the applications, the AI leaderboard collapsed. Six of the AI judge's top 10 tools were rated weak by humans. Even more telling, the application that won the entire competition placed 19th on the AI leaderboard.

“The ratings had a pretty poor correlation with the AI judge,” swyx said. “We want people who understand the job to be done instead of this specific spec, and instead of mechanically cloning a product you can actually define what a good user experience looks like and step into the role of our real testers.”

LLM judges reward mechanical parity. If a test scenario checks whether an admin can invite a speaker, an agent gets full points for a raw form that fires a database record, even if the flow requires four confusing screen hops. One builder, Open Stash, pointed out that the features they were most proud of were completely omitted from the automated eval. The AI judge had no way to register smooth workflows, clean layout hierarchy, or defensive edge-case handling. It only verified the prompt's checklist.

Automated evals create good software only when the metric captures the entire value exchange. When you eval SaaS via isolated model assertions, you optimize for ticking boxes while ignoring whether a human would tolerate using the interface every day.

What to Do With This

Audit your internal agent evaluation suite tomorrow morning. Take five user flows that your test runner rates as a perfect pass, hand them to a designer or support lead with zero context, and watch where they stall. If your team treats green CI runs from an LLM judge as proof of product quality, recalibrate immediately by tying your release gate to unassisted human task completions.