Key Takeaways

  • Warp dropped its internal cost per pull request from $80 to between $15 and $20 by scoring live agent sessions and routing tasks dynamically.
  • Coding agents systematically bloat repositories by writing throwaway test suites during debugging and leaving them committed in the codebase.
  • Background LLM judges evaluate agent execution traces in real time, scoring each session against concrete failure patterns like token waste or redundant logic.
  • When scoring flags clustered issues, such as 54 triage failures traced to a single agent getting stuck, an automated improvement agent writes a pull request to update the factory code.

The Method

Most teams treat coding agents like contractors. You hand them a prompt, wait five minutes, review a messy pull request, and fix their mistakes by hand. Zach Lloyd took a different path at Warp. Instead of letting agents run untracked, Warp turned its engineering pipeline into declarative code evaluated by automated judges.

Lloyd calls these background evaluators scorers. As he explained: “The final thing that's really cool, once you have this sort of closed-loop system going, is that you can actually look underneath the hood at how well your coding agents are doing along different dimensions. And so, you can think of this as like online evals or we call it scorers.” (play from 9:26)

These scorers search for silent failure modes that pass continuous integration but quietly damage a software product. One common target is test pollution. When an agent attempts to solve a hard ticket, it often writes temporary test suites to verify its internal logic. Once it gets the green checkmark, it leaves that scaffolding behind. As Lloyd pointed out: “This bloats your codebase. It wastes tokens.” (play from 11:05) A background judge flags the redundancy immediately.

Once scoring data accumulates, Warp runs a second layer of automation. Instead of asking human managers to read eval logs, background improvement agents analyze trace failures and alter the pipeline itself. Lloyd described an exact production scenario: “Our self-improvement agents were looking at how well our triage step was doing. We have a triage step, and they're saying, 'Okay, we have 54 failure cases of this particular form, where it would have been better if our triager had been able to spawn a subagent to do its investigation.'” (play from 14:55)

Because Warp writes its agent pipeline, tool definitions, and repository links as declarative code, the self-improvement agent does not merely suggest a fix. It writes code to update the triage step, tests the change, and opens a pull request against the pipeline repository. The factory modifies its own engine.

Where This Breaks Down

This loop breaks down if your pipeline lacks deterministic validation. If an agent writes a pull request to alter how other agents behave, you need rock-solid integration tests for the factory itself. Without strict guardrails, an auto-improvement agent can overfit to a temporary cluster of weird user tickets and rewrite prompts in ways that break simple downstream tasks.

It also fails if you run small volumes. If your team only ships ten pull requests a week, an online scoring agent burns more money in judge tokens than it saves in engineering hours. Warp saw their cost drop from $80 to $15–$20 per pull request because they process massive volume. At that scale, eliminating redundant unit tests and cutting token bloat yields compounding financial returns.

What to Do With This

Audit your last 20 agent-generated pull requests tomorrow morning. Tag every test file that was added strictly to verify intermediate assumptions rather than production requirements. If more than three pull requests contain dead test scaffolding, write a post-generation prompt rule that bans agents from staging test files unless they match a specific acceptance criterion in the original ticket.