Key Takeaways

  • Emmanuel Ameisen rejects the phone keyboard autocomplete mental model because predicting the next token in hard tasks requires modeling underlying real-world systems.
  • In poetry generation tests at Anthropic, models internally chose rhyming words like "rabbit" multiple tokens before writing them.
  • Models do not need to write out their reasoning in text to plan ahead; their intermediate layer activations store and execute multi-step strategies silently.
  • Apparent sudden jumps in AI capability stem from smooth underlying scaling curves that cross thresholds where internal plans suddenly become visible.

Next-Token Prediction Forces Models to Plan Ahead

People still describe large language models as glorified autocompletes. You type a prompt, and the network guesses the next word the same way a phone keyboard suggests words above your thumb.

Emmanuel Ameisen, an interpretability researcher at Anthropic, argues that this mental model is flawed. The training objective is simple: guess the next word. But mastering that objective across complex human language requires something far deeper than surface-level pattern matching. To predict the next token in an argument, a proof, or a story, a network must build an internal model of the world.

As Ameisen explained to Lenny Rachitsky: “As these models get better at the seemingly simple task, the sort of best way to solve it, at least for us, is to just kind of understand the underlying structure of the world. And in fact, some of the stuff that we look at inside model shows that they do understand that, and they do sort of manipulate these concepts like we do.”

Hidden Activations Show Silent Commitments to Future Words

The clearest evidence against the shallow autocomplete theory comes from looking directly inside neural networks. Ameisen and the interpretability team at Anthropic inspect intermediate layer activations while models run.

In one experiment, the team examined how models write rhyming poetry. If a model were merely stringing words together based on recent context, it would generate a line, reach the end, and then search for a word that matches the rhythm and rhyme of the previous line.

That is not what happens. Ameisen observed that inside intermediate results, the model commits to a target rhyme well before reaching the end of the line.

“That means that in practice, we find that models will like in their intermediate results sometimes formulate a plan,” Ameisen noted. “They write a line and then like in their intermediate results, they're like, 'Oh, oh, oh, this is an it rhyme. I should rhyme with rabbit.' And then every future word that they predict looks back to that plan they made.”

The model never writes "rabbit" in an external scratchpad. It outputs no reasoning trace. The commitment exists entirely inside the intermediate activations across hidden layers. Every preceding token selected by the model is shaped by that silent destination.

“And you can do that without saying the plan,” Ameisen said. “That's the cool part is like this plan that we find is in these intermediate results inside the model, but the model doesn't even have to sort of like say it out loud.”

What to Do With This

Stop assuming your model cannot plan unless you force it to write out every step. Run an A/B test on your most complex prompt this week: test your verbose chain-of-thought prompt against a direct prompt that specifies only the required constraints and end state. Measure whether the extra generated tokens actually improve accuracy or simply inflate your latency and API costs.