Key Takeaways

  • Public coding benchmarks like SWE-bench test generic Python problems, not the custom frameworks, internal packages, or API boundaries of your production codebase.
  • Zach Lloyd cut Warp's internal cost per pull request from $80 down to $15 to $20 by routing tasks based on internal repo benchmarks instead of public leaderboards.
  • Reliable benchmarking requires varying both the underlying frontier model and the agent harness wrapping it.
  • Building an internal evaluation suite starts by capturing real engineering runs, curating them by task type, and replaying them across a configuration matrix.

The Flaw in Public Leaderboards

Engineering teams pick foundation models based on public eval tables. They see a frontier model score high on SWE-bench and assume it will write clean code inside their private monorepo. Then they deploy it, watch it hallucinate internal utilities, and wonder why their engineers spend hours correcting automated pull requests.

Lloyd points out that public tests operate in a vacuum. A model that solves open-source bugs often stumbles when confronted with proprietary design systems, legacy configurations, or custom microservice wrappers. As Lloyd explained to swyx:

“And so what our opinion on this is and what the the world sort of needs is like out of the the realm of vibes for understanding these coding agents and actually into the world of measurement.” (play from 13:36)

Measuring agents on vibe checks or public leaderboards burns cash. Warp saw this directly in their own systems. Early iterations of their coding agents cost roughly $80 per pull request because they directed every task to the most expensive frontier model without checking whether a lighter model inside an optimized harness could do the job.

Build an Internal Replay Suite

Instead of trusting external metrics, Lloyd advocates turning your actual developer traffic into a benchmark. When an agent attempts a task inside your codebase, capture the input, the environment, and the final result.

Lloyd describes the setup:

“And so, benchmarking in this context, in the factory context, means essentially the ability to, like, take any coding agent that's running your system. You can sort of add it to a benchmark suite, which means, like, you know, you might curate a set of coding agent runs that were related to, like, doing front-end coding or something like that.” (play from 16:14)

Once you store those task samples, you can run experiments across two axes: the model and the agent harness. Most teams only swap the model, leaving system prompts and tool definitions static. Lloyd found that testing different agent harnesses against the same model yields massive differences in accuracy and latency:

"And then once you have that, you have the ability to sort of replay those same tasks against a matrix of different configurations. And so you know, I'm a big believer in not just allowing you to vary the model, but also to vary the harness. And when you do something like this, what you end up getting is a result that looks very similar to like the published results for SweetBench or CriminalBench, but on your company's own real runs."

Route Workflows by Correctness and Cost

The goal of an internal benchmark is not bragging rights. It is automated routing. Once you know which model and harness pairing solves front-end tasks reliably, you stop sending routine CSS or component updates to the most expensive model on the market.

Lloyd notes the direct outcome:

“And then the cool thing about this is you can take this information and feed it into a model routing strategy, or harness routing strategy, honestly, that's defined specifically for your own workflows and code base.” (play from 17:46)

By testing historical tasks against a matrix of model and harness pairings, Warp shaved pull request costs down to $15 to $20. They stopped paying top dollar for tasks that smaller models could complete without error when equipped with the right tooling.

What to Do With This

Audit your team's last 20 automated agent runs this week and pull five that passed cleanly and five that failed. Save the initial prompts, repo states, and diffs into a private test directory. Before you upgrade your team to a new flagship model, run those 10 tasks through both the old and new setups to see if accuracy actually changes on your own code.