Why Frontier AI Moats Are Evaporating
John Schulman and Beren Millidge explain how router data and easy RL distillation strip frontier AI labs of long-term software moats.
AI researchers debate how close we are to recursive self-improvement
Dwarkesh Patel hosts AI researchers John Schulman, Charlie O’Neill, and Beren Millidge to debate the frontier of artificial intelligence and recursive self-improvement. They discuss the limits of transformers and reinforcement learning, the mechanisms of model distillation and continual learning, and provide timeline predictions for drop-in remote workers and superhuman AI research.
Key takeaways
- Post-training reinforcement learning produces very few actual information bits, making top capabilities trivial to distill once seen in public API outputs. Read more →
- Iterative fine-tuning on live user traces causes catastrophic forgetting after hundreds of micro-updates, erasing base capabilities. Read more →
- Charlie O'Neill estimates autonomous remote workers arrive in roughly one year if granted direct programmatic tools like Slack bots, or two years if restricted to human browser interfaces. Read more →
- Beren Millidge points out that poor sample efficiency forces labs into sim-to-real training, because models currently demand thousands of human interactions to master complex tasks. Read more →
- Charlie O'Neill questions whether the current recipe of transformers and reinforcement learning can discover major algorithmic jumps or reach the theoretical optimum of a learner on a chip. Read more →
- Mid-training on synthetic reasoning datasets does roughly 80 percent of the work before reinforcement learning begins. Read more →
6 articles from this episode
Why Base Models Can't Learn Continuously in Production
Charlie O'Neill and Beren Millidge explain why live weight updates fail and why labs must retrain base models from scratch.
When Drop-In AI Remote Workers Actually Arrive
AI researchers Charlie O’Neill and Beren Millidge predict drop-in remote workers in 1 to 3 years and 10x research uplift in two.
Why Real-World AI Agents Won't Come From Sandboxes Alone
AI labs bet on simulated RL, but Cursor proved user heuristics win. Here is how frontier models must adapt to real deployment.
Why Recursive AI Self-Improvement Will Stall Before 2036
John Schulman, Charlie O'Neill, and Beren Millidge debate the architectural and verification bottlenecks blocking autonomous superintelligence.
Why Reinforcement Learning Scales Reasoning Models
Beren Millidge and John Schulman explain how one bit of RL feedback and synthetic mid-training scale LLM reasoning.
More Dwarkesh Podcast episodes
- Ajeya Cotra – Inside the OpenAI agent swarm that hacked Hugging FaceSep 6 · 4 articles
- The OpenAI/Hugging Face attack, clearly explainedSep 6 · 5 articles
- Ryan Greenblatt – What happens once AI can automate AI research?Aug 16 · 3 articles
- General relativity from first principles – Adam BrownJul 12 · 2 articles