Why Recursive AI Self-Improvement Will Stall Before 2036

Key Takeaways

• Charlie O'Neill questions whether the current recipe of transformers and reinforcement learning can discover major algorithmic jumps or reach the theoretical optimum of a learner on a chip.
• John Schulman points out that recursive improvement bottlenecks wherever models show poor judgment or fail to check their own work accurately.
• Beren Millidge argues that automated recursive loops require models to propose and evaluate their own objectives over long horizons without drifting off course.
• Schulman predicts that the final durable role for human workers will be defining alignment objectives and specifying human preferences.

The Disagreement: Architectural Limits vs. Objective Drift

The race to reach superintelligence by 2036 assumes AI will eventually improve its own code in an endless, compounding loop. But researchers disagree on what snaps that chain first.

Charlie O'Neill targets the algorithmic architecture itself. He doubts the current mixture of transformers and reinforcement learning can escape local optima. “For me, it’s a question of how far off the global optimum of 'a learner you could have on a chip' is from the transformer + RL, basically the current recipe,”▶ 2:49 O'Neill explains. If transformer architectures cannot invent radically better learning algorithms, automated AI research hits a hard ceiling.

Beren Millidge and John Schulman see a different barrier: verification and goal setting. Even if an architecture has room to grow, an autonomous system must supervise itself.

Schulman argues that progress stalls when self-evaluation fails: “Each time a new model comes out, it’ll catch up in some of these areas. But you end up getting bottlenecked by the places where the model is weaker and where it has worse judgment, or the models can’t check themselves well enough.”▶ 1:51

Millidge pushes this further to autonomous goal selection. “To have any kind of self-propelling automated loop, we need the AI to propose objectives, optimize them, figure that out, propose a new objective, and have this not go off the rails at any point for a long, long time,”▶ 13:40 Millidge notes. Without true generalization, systems merely master benchmark suites. “If somehow that continues, and there’s never the true spark of generalization that occurs, I think that could lead to the AI just being extremely good at everything that people put into a benchmark or put into an environment.”▶ 1:13

Who's Right (and When They're Wrong)

O'Neill is right if algorithmic discovery requires genuine leaps outside transformer mechanics. Current reinforcement learning optimizes known reward functions over existing token spaces. It does not easily invent entirely new computational structures. If reaching superintelligence demands a post-transformer breakthrough, current models cannot bootstrap their way there alone.

Schulman and Millidge are right on shorter operational horizons. Before you worry about whether an AI can invent a new architecture, you will hit the evaluation wall. Automated self-improvement fails early because models reward their own clever mistakes. When an AI generates both the code and the evaluation metric, errors compound across generations.

This makes Schulman's observation practical: “I would say that the last job for humans, or the role for humans that’ll last the longest, is defining the objective and deciding what we actually want.”▶ 15:45 Until models can verify open-ended reasoning without human ground truth, human judgment remains the primary anchor.

What to Do With This

Stop treating AI agents as fully autonomous research loops. Tomorrow, audit your automated evals and agent workflows: identify every step where a model grades its own output. Replace those self-grading nodes with deterministic unit tests, external compilers, or strict human review checkpoints to prevent silent metric drift.