Key Takeaways

  • When enterprise software vendors quoted $43,000 annually for speaker management software, the AI Engineer team ran the Kill My SaaS competition, generating 72 entries averaging 91,000 lines of code.
  • Winning builders proved autonomous coding agents fail when treated as chat-based code generators; high-output builders spent hours planning schemas and scraping documentation before writing prompts.
  • Builder Marko Kraemer spent eight to nine hours mapping database schemas and business logic in pseudocode before opening an agent tool, while Yazin used Firecrawl to extract Sessionboard's documentation.
  • Finalist Valentin De Matos parsed YouTube product demos frame by frame using Codex to reconstruct missing interaction patterns and visual flows.
  • To prevent synthetic evaluations from masking broken user experiences, finalists relied on Valentin De Matos's 5-Minute Usability and Trace-Debugging Loop.

The Valentin De Matos 5-Minute Usability and Trace-Debugging Loop

When enterprise software vendors quoted the AI Engineer team $43,000 annually for speaker management software, the team launched the Kill My SaaS competition. Across 72 submissions averaging 91,000 lines of code, one lesson stood out: conversational coding fails on large applications. Automated checks passed code that human users found unworkable.

The finalists who built functioning software treated prompting as the final stage of development, not the starting line. Builder Marko Kraemer spent hours mapping foundations first: “If the UX UI is done, the only thing I do is I sit down for 8 or 9 hours and in the most granular way I plan out the entire data schema, the business logic and literally pseudo code.” Yazin scraped public knowledge bases: “I took the Google doc, I turned it into a markdown file and then I also ran Firecrawl on Sessionboard's help portal as well as a few other sites that were similar.”

Valentin De Matos went a step further, scraping video walkthroughs: “Basically, I just asked Codex, hey dude, you must be able to draw me every single flow of every single feature of this app. And amazing thing is that was a bunch of YouTube videos. And so he took screenshot frame by frame to be like, 'oh, okay, here's what happened.'”

Host swyx noted the pattern: “Notice that nobody said they typed one prompt and walked away. Really the prompt was almost the last step that you lead up to after building up a full knowledge of the problem, mocks, docs, and specs.”

To ensure agents produced usable software instead of code that merely compiles, De Matos deployed a five-step quality loop:

  • Step 1: 5-Minute Operator Persona Emulation: Step directly into the end user's shoes with a strict 5-minute time constraint to click through and complete primary user flows.
  • Step 2: Smoothness Gate: If the 5-minute test is not completely smooth, halt further progress and diagnose why the workflow broke (missing context, unclear expectations, or lack of local state testing).
  • Step 3: Trace Exploration: Inspect agent execution traces to pinpoint root causes rather than manually patching styling or logic errors directly.
  • Step 4: Research-Injected Prompt Revision: Instruct the agent to perform domain research defining what makes a good version of the specific broken workflow before re-generating the step.
  • Step 5: Full Workflow Re-run with Local Execution: Rerun the entire flow end-to-end, enforcing that the agent boots and validates against the local running application.

When This Works (and When It Doesn't)

This framework works when building software that human beings operate by hand, where small interaction flaws ruin the experience long before automated unit tests catch a bug. Automated AI evaluators will happily pass an app that returns a 200 HTTP status code, even if a user has to reload the page three times to submit a speaker bio.

This loop breaks down on pure backend infrastructure or headless data pipelines without user interfaces. If you are tuning database migrations, event streaming throughput, or network latency, spending five minutes clicking through a screen wastes time. In those environments, deterministic test suites and strict integration assertions outperform manual persona runs.

What to Do With This

Stop prompting agents to build full features from high-level specs. If you are cloning an internal SaaS tool this week, test your agent's current output as an operator for exactly 300 seconds.

If you hit friction, like an awkward speaker acceptance modal that drops form state upon refresh, do not manually patch the code. Instead, open the execution trace, identify where the agent missed state persistence, and command it to research how modern web apps structure multi-step form state before it rewrites the component. Once it finishes, boot the local build and run the 5-minute test from the beginning.