Key Takeaways

  • Engineering teams run into a blind spot when developers operate isolated instances of Claude Code, Cursor, or Codex in local terminals.
  • Centralizing agent runs lets organizations capture full transcripts across tools, from initial Slack threads to Linear tickets and GitHub pull requests.
  • Tracking human intervention frequency per merged pull request shows the true level of automation across an engineering org.
  • Warp dropped its internal cost per pull request from $80 down to $15–$20 by centralizing runs, evaluating agent traces, and applying model routing.

The Blind Spot Inside Local Terminals

Most software teams think they are adopting AI quickly. In practice, they have created a sprawling collection of disconnected silos. One engineer runs Claude Code in their terminal. Another uses the Codex app. A third experiments with Cursor.

Each developer gets a local speed boost, but the engineering organization learns nothing from it. When an agent fails, only that engineer sees the error. When an agent succeeds, nobody else gets to study the prompt or the path it took. The company has zero visibility into what works.

Zach Lloyd noticed this disconnect while examining how engineering teams build with agents: “On your team today, what we when I talk to companies for the most part, they don't have something like this because most of their engineers are like individually using Cloud Code in their terminal or Codex the Codex app or something like that in their own local setups. But, they want to get to a world where they have this centralization.” (play from 8:05)

Without centralization, you cannot measure baseline engineering health. You cannot see whether an agent saved two hours or created four hours of messy code review. You cannot calculate what you spend per pull request.

The Hidden Lifecycle of a Pull Request

Code does not start or finish inside an editor. A feature request begins in a Slack conversation, turns into a Linear ticket, moves to a repository branch, triggers continuous integration checks, and goes through review comments on GitHub.

If your agent tooling only lives in an engineer's personal terminal window, you lose the connective tissue. Lloyd explains the reality of these workflows: “You need a record of all of the interactions from all of the tools to get to a result. And so what I mean by that is like you might have a factory conversation that starts in Slack and then continues into linear, and then continues into GitHub, and there's a code review comment, and that whole interaction is going to result in some piece of merged code.” (play from 21:15)

Centralizing agent execution turns those fractured touchpoints into a unified record. When you store the full trace of an agent's run across every tool, you can finally evaluate the system. That evaluation is what allowed Warp to reduce its internal cost per pull request from $80 down to between $15 and $20. They used trace logs to guide model routing and test harnesses against real production tasks.

Measure Human Touches Instead of Token Counts

Token consumption and latency give you infrastructure metrics, but they do not tell you if software delivery improved. Fast code generation is useless if an engineer spends half their day fixing syntax errors and re-running failed test suites.

Instead of counting tokens, track human touch points. Lloyd points to a direct measure of efficiency: “And so, that's things like, you know, traditional Dora metrics, how long does it take to ship something? It's things like how automated are you? And so, we have a sort of metric around like, for every PR that gets merged, how many times did a human have to do something with it? What's your throughput?” (play from 8:45)

Every time an engineer has to correct an agent's plan, fix a hallucinated dependency, or manually clean up a diff, log that event. If a pull request requires eight manual interventions before it merges, the agent did not automate the task. It created busywork. A centralized trace system turns those interventions into concrete training data, showing your team exactly where the tooling breaks down.

What to Do With This

Audit your team's last ten merged pull requests that used coding agents. Count how many manual interventions each pull request required, including code review comments, terminal fixes, and rejected diffs. If you cannot track those handoffs across Slack, Linear, and GitHub, set up a shared log for agent transcripts before paying for more individual seats.