AI Benchmarks Lie: Compute, Not Model, Drives Results
OpenAI's Noam Brown argues traditional AI benchmarks misleadingly ignore test-time compute. Learn why this oversight distorts model capabilities and how to evaluate smarter.
Why Traditional Benchmarks Fail Modern AI Models with OpenAI Research Scientist Noam Brown
With Sarah Guo, Noam Brown · Sunday, June 28, 2026
This episode features OpenAI research scientist Noam Brown discussing the shortcomings of current AI model evaluation benchmarks, particularly their failure to account for large-scale test-time compute. He explains how this oversight impacts the assessment of model capabilities and has significant implications for AI safety and responsible scaling policies. Brown also shares insights into the true nature of recursive self-improvement and the potential of latent capabilities in current models.
Key takeaways
- Current AI model benchmarks, often presented as a single-point "grid," fail to account for the amount of compute spent during evaluation, masking true model capabilities. Read more →
- Early LLMs (pre-GPT-5.2) were useless for complex reasoning tasks like building a poker bot; they simply couldn't get started. GPT-5.2 could help, but required constant oversight. Read more →
- Your current AI models, even public ones like GPT-5.5, hold significant untapped "latent capabilities." OpenAI recently disproved the complex Erdos unit distance conjecture internally, a feat Noam Brown believes GPT-5.5 could mimic with roughly $100,000 in dedicated compute. Read more →
- Forget the science fiction fantasy: Brown says we're not heading for an "overnight intelligence explosion" where AI instantly becomes superhuman across the board. Read more →
4 articles from this episode
Your LLM Is a 100x Optimizer, Not a Novelist
OpenAI's Noam Brown used GPT-5.5 to build poker bots 100x faster, showing LLMs excel at optimizing existing solutions, not inventing novel ones.
Stop Waiting for GPT-X: Your Current AI Has Hidden Power
OpenAI's Noam Brown reveals why founders overlook massive latent AI capabilities in models like GPT-5.5. Unlock them now, don't wait.
No Overnight Explosion: AI's Recursive Self-Improvement is Bottlenecked by Time
OpenAI's Noam Brown reveals AI's 'fast takeoff' is a myth. Current models are bottlenecked by test-time compute, making progress gradual.
More No Priors episodes
- Frontier Chips for Frontier AI Labs, with Walter Goodwin, Founder/CEO of FractileOct 4 · 5 articles
- Re-Founding Incumbents for the AI Era with Sequence Holdings Co-Founder and CEO Michael LeeSep 27 · 5 articles
- Coinbase’s Everything Exchange: Agentic Finance, Stablecoins & Tokenization with CEO Brian ArmstrongSep 13 · 6 articles
- Redefining Chip Architecture with Arm CEO Rene HaasSep 6 · 7 articles