Key Takeaways
- Public AI benchmarks have detached from software reality over the last nine months because labs face perverse incentives to inflate paper scores.
- Diogo Almeida separates the industry into truthful Tier 1 labs (OpenAI and Anthropic) and Tier 2 contenders (including Google, Meta with Llama 4, and open-source Chinese labs) who game evaluations to defend valuations and internal morale.
- Non-technical buyers and enterprise executives suffer the most from benchmark manipulation because they lack the time to run hands-on evals in production.
- Anthropic models consistently outperform competitors in actual developer workflows despite often matching or trailing other frontier models on public scorecards.
The Incentive to Lie Starts at Tier 2
Every enterprise software buyer wants a single score to justify a million-dollar model decision. Labs know this, so they optimize for public tests instead of real software execution. Diogo Almeida, co-founder and CEO of TypeSafe AI, argues that this behavior splits the market right down the middle.
“The top tier labs are incentivized to be truthful,” (play from 43:23) Almeida says. “So I believe that OpenAI and Anthropic are mostly truthful. The second tier labs are incentivized to lie to themselves.” (play from 43:27)
Frontier leaders like OpenAI and Anthropic already own developer distribution. They do not need to juice an academic test to raise their next equity round or convince researchers to stay. The pressure falls entirely on everyone else: well-funded contenders trying to prove they belong in the same sentence as the market leaders.
Almeida points directly to Big Tech and open-weight players falling into this trap: “This is what happened with Llama 4 where they were actually too on the nose about benchmark maxing even though there's so much internal incentive to do that. Same thing for Google right now and same thing for a lot of the other Chinese labs.” (play from 43:49)
When a model's public score becomes its primary marketing asset, engineering teams end up training directly against test sets. The result looks brilliant on a PDF chart and falls apart the moment a developer asks it to write clean, multi-file code.
Anthropic Wins Where Leaderboards Fail
Evaluating raw intelligence remains an unsolved problem. “I think people have a hard time understanding intelligence as a product,” (play from 20:25) Almeida says, “which is why people try to use benchmarks as band-aids.” (play from 20:30)
A spreadsheet metric cannot measure whether a model respects system instructions, avoids hallucinating edge-case syntax, or fails gracefully when an API schema changes. Because of this gap, real developer sentiment has drifted completely away from public rankings.
Over the last nine months, engineers who actually ship production software realized that Anthropic models deliver better output regardless of what leaderboard tables claim. “Benchmarks don't mean anything,” (play from 20:46) Almeida notes. “Anthropic models are just the best even if the Chinese models match it in benchmarks.” (play from 20:48)
This gap creates an information tax on teams that rely on external reports rather than hands-on testing. “All benchmarks are bad. There's perverse incentives,” (play from 42:29) Almeida warns. “And ironically, the people it hurts most are the people who do not have time to actually play with the models because they're the ones who get manipulated by the benchmarks themselves with higher stakes.” (play from 42:34)
What to Do With This
Strip every public benchmark score from your internal AI evaluations this week. Pick 50 messy, production-grade tasks your application failed on last month, run them directly through Claude 3.5 Sonnet, GPT-4o, and your candidate open-weight model, and score the outputs on real execution success rather than leaderboard claims.