Key Takeaways
- Brute-force pre-training hit diminishing returns around GPT-4.5, ending the 2019 to 2023 era where adding raw compute reliably produced proportional model gains.
- Task definition determines capability: Diogo Almeida argues that a GPT-2-sized model built on the correct task framing can outperform a theoretical GPT-9 trained on generic objectives.
- Enterprise data lakes offer little value for frontier model training because historical corporate logs do not contain the on-policy, interactive decision data needed for reliable automation.
- Reinforcement learning from human feedback worked because researchers designed a new task around preference ranking rather than discovering an algorithmic trick.
- Teams escaping the compute plateau organize their work around Almeida's Bitterest Lesson Hierarchy.
The Bitterest Lesson Hierarchy
Rich Sutton's classic Bitter Lesson stated that general methods leveraging compute beat human-designed algorithms every time. Almeida argues that this principle governed the scaling era from 2019 to 2023, but broke down as models reached the scale of GPT-4.5. “We kind of stopped scaling as a field because it stopped having a payoff,” (play from 5:19) Almeida noted. In its place, he proposes an updated priority stack for building capable intelligence:
Component 1: Doing the Right Task (North Star)
Define the exact objective structure and framing of what you want the model to accomplish. Without the right task (for example, human preference comparison in RLHF or latent chain-of-thought in RLVR), the necessary training data does not exist anywhere in the world and cannot be mined.
Component 2: Tailored Data
Curate and generate on-policy, in-distribution data tailored precisely to push the frontiers of the chosen task. Data quality and task alignment matter many times more than raw computational scale.
Component 3: Targeted Compute
Apply compute to scale the training and inference of the data-backed task. Compute is necessary for execution but experiences sharp diminishing returns when scaled without task-specific data.
Component 4: Underlying Algorithms
The specific underlying model architecture and training algorithms matter least. An optimal task framing and data pipeline on an older model class will outperform a superior algorithm trained on generic objectives.
When This Works (and When It Doesn't)
This hierarchy works whenever you build systems that require high reliability in specific operational domains. If you want a model to execute legal compliance reviews or automate backend database migrations, throwing more GPUs at generic next-token prediction will fail. Success requires defining what a completed, verified migration step looks like, generating synthetic state trajectories, and fine-tuning an existing model on those traces.
The framework breaks down during the earliest exploratory phases of general model development. If you do not yet understand the capabilities of a new architecture, you cannot construct the synthetic task environments or feedback loops that generate on-policy data. In those specific exploratory windows, broad pre-training across broad corpora remains the only practical way to discover what base models can do.
What to Do With This
If you run an AI engineering team, cancel your plan to fine-tune an open-source model on your customer support ticket history this week. Almeida points out that enterprise data lakes rarely help because “it's probably not shaped the right way.” (play from 12:00) Historical support tickets show you what humans wrote in 2022; they do not show the stepwise decision paths, tool calls, and error corrections required for an agent to solve tickets autonomously.
Instead, spend the next three days defining the verification environment. Write down the five exact success criteria for a single workflow, build a simulator or sandbox where an agent can take actions, and record the failure points. Once the task is defined and you can generate trajectory data from those specific mistakes, you can train a tiny model to beat a giant, generalized model on that exact workflow.