Key Takeaways

  • Academic benchmarks saturate at 99% to 100% accuracy within weeks of release, forcing builders back to subjective vibe checks.
  • Raw token pricing charts are misleading: a model like 3.8 Flash looks 13 times cheaper per token than Astra, but Astra costs less per finished task.
  • Steven Heidel observed that reasoning models need far fewer prompt iterations and generated tokens to complete complex multi-step workflows.
  • The best evaluation test is checking for Gell-Mann amnesia: assign the model a workflow you understand intimately and see if the output withstands expert scrutiny.

The Breakdown of Standard AI Leaderboards

Every time researchers publish a new test suite, frontier labs hill-climb the dataset until their models score near perfection. As John Coogan explained, “the benchmark era might be coming to a close in some in some way because as soon as the benchmarks created, it's hell climbed very very quickly and then it's at 99 or 100% pretty quickly and everyone sort of their eyes start glazing over in benchmark discussions and it just goes back to like the vibe check.”

Even industry insiders mock the endless stream of chart victories. Jordi Hays highlighted Jack Altman's reaction to recent scoreboard claims: “Jack Altman saw the benchmarks and said, 'Sam, exclamation point. I guess even a blind squirrel can find a nut sometimes. Congrats.'”

When every leaderboard displays green checks across the board, standard metrics stop helping engineers decide what to put into production. A test score does not reveal whether a model can navigate a messy internal codebase or handle ambiguous instructions in a production environment.

Why Token Pricing Charts Lie

Software teams still calculate their AI unit economics by looking at per-million token rates on vendor pricing pages. That approach gives false cost estimates.

Coogan stated the reality bluntly: “Token pricing is effectively meaningless now. The benchmarks are meaningless. The token pricing charts are meaningless.” He pointed to Steven Heidel's analysis of model economics to explain why: “3.8 flash looks 13 times cheaper when measured per token, but Astra is cheaper per task since it's far more efficient. Measure your cost per task, not per token.”

A cheaper model often burns thousands of tokens looping through failed attempts, incorrect tool calls, and self-correction steps. A more capable reasoning engine might cost ten times more per token, but it solves the problem in a single pass using a fraction of the context window. Evaluating raw token costs without measuring task completion rates causes engineering teams to pick slow, expensive models disguised as budget options.

The Gell-Mann Amnesia Evaluation Test

If public benchmarks and token calculators fail to tell you which model to deploy, you need a reliable filter for your own evaluations. Coogan suggests testing against human expertise:

“You basically want to test if Gell-Mann amnesia is triggered by you using the model. So point it at something that you intimately understand and see if you're impressed.”

The Gell-Mann amnesia effect describes reading a newspaper article about your own field, spotting obvious errors, and then turning the page to read international news assuming the rest of the paper is completely accurate. When testing a model on generalized tasks, you project competence onto outputs you cannot independently verify. When you force it to write code in your specific framework or handle domain math you know by heart, the facade drops immediately.

What to Do With This

Pick your top production prompt workflow tomorrow morning. Run 50 real production inputs through your current model and through a frontier reasoning model, tracking two numbers: total dollar spend and clean completion rate without manual intervention. Divide the total dollar cost by successful runs to find your real cost per task. If the expensive model completes the job in fewer attempts for a lower net cost, swap your production routing immediately.