LLMs Run Internal Gut Checks While They Reason

Key Takeaways

• Emmanuel Ameisen at Anthropic studies internal neural representations to see how models make decisions under the hood.
• LLMs develop separate internal "gut-check" circuits that evaluate plausibility in parallel to calculation, such as instantly flagging that 4 + 5 cannot equal 23 without doing the formal math.
• Single-token classifications act like Daniel Kahneman's System 1 intuition, while multi-token chains mirror System 2 deliberative thought.
• Interpretability tools prove that when prompted to perform a surface task while keeping an alternate concept in mind, models maintain intermediate representations of the secondary task in latent space.

Models Check Their Own Work in Real Time

When you ask an LLM to solve a multi-step problem, human intuition assumes the model reasons in a single linear sequence. Anthropic researcher Emmanuel Ameisen found that the internal mechanics tell a different story. As Ameisen studied how Claude models sample tokens, interpretability research revealed two distinct cognitive engines running at once.

One engine performs step-by-step reasoning. The other operates like an instantaneous sanity check. Ameisen explained that models will build intermediate steps, but they also maintain “a separate mechanism which is kind of like a gut check... as they reason they'll have like some part of them that will flag like something's wrong.”▶ 1:02:23 For instance, if an equation proposes that 4 + 5 equals 23, the model's gut-check circuit immediately tags the result as impossible before completing formal arithmetic steps.

This mirrors Daniel Kahneman's model of human cognition. Kahneman divided thinking into System 1 (fast, reflexive, intuitive) and System 2 (slow, deliberate, analytical). In artificial intelligence, builders often associate System 2 with long reasoning chains and chain-of-thought scratchpads. But Ameisen points out that rapid single-token models, such as Jev, function like pure System 1 intuition: “You're not letting the model sample and reason. It's just your gut feeling, is the answer A or is the answer B.”▶ 1:00:08 Instead of executing pure arithmetic, LLMs develop high-speed heuristics that supervise their own calculations.

Latent Space Can Run Dual Tracks

The existence of separate heuristic circuits points to a broader reality inside large models: internal activity does not match external text generation. Builders often treat token output as the complete record of what a model is processing. Ameisen's interpretability work proves that models can juggle hidden agendas in their latent space.

Researchers tested this by prompting a model to execute a specific surface task while holding a completely different thought in reserve. Ameisen noted: “You can tell the model, hey, do this task and think about this other thing and then you can use interp to see if like there's intermediate representations that map to the other thing and that works. You sort of see that it does think about the other thing.”▶ 1:02:56

The model executes the surface prompt cleanly, but its hidden activation layers simultaneously maintain representations for the secondary assignment. This reveals that models possess internal bandwidth beyond what shows up on your screen. It also exposes the limits of evaluating AI agents solely by reviewing their final written logs. A model can satisfy external requirements while its intermediate vectors track unprompted patterns, drift, or constraints.

What to Do With This

Audit your agent evaluation pipeline this week. Stop relying solely on long chain-of-thought outputs for verification, and start testing zero-generation sanity checks. Route your model's draft answer through a single-token binary classifier prompt asking: "Is this output physically possible: Yes or No?" If the rapid System 1 classifier returns "No," intercept the run before spending compute on further downstream calls.