Key Takeaways
- Training an AI model produces capability without explanation. As Emmanuel Ameisen puts it, “to make them is not to understand them.”
- Frontier models develop internal representations that hold across formats: an image of an eye, the word eye in any language, and an SVG line drawing of an eye activate the exact same internal vector inside Claude.
- Interpretability treats artificial intelligence like biology rather than standard software engineering, studying emergent behaviors inside intermediate computational layers after training finishes.
- Anthropic tests whether an internal pattern actually drives behavior using Anthropic's Intermediate Activation Interpretability Method, directly overriding intermediate numerical values during generation.
The Anthropic's Intermediate Activation Interpretability Method
Step 1: Record Activations Across Arbitrary Data
Pass a bunch of text, just all of the text that you can find, just random text through the model, and you record these intermediate numbers for every example.
Step 2: Decode Concepts and Look for Statistical Patterns
Just try to see if there are patterns: are there numbers, pairs of numbers, or specific series of numbers that only occur when the model is processing English text, French text, text about love, or text about eyes or whatever.
Step 3: Test Hypotheses with Targeted Prompts
We can do a bunch of experiments: we can ask it harder questions and then label all of the intermediate states and see, oh this is what it's doing.
Step 4: Causal Activation Steering
We can even go inside its activations and change it, and then see what would happen if, while the model is writing these intermediate results, I went and secretly changed one of the results, and does that do what I would expect?
When This Works (and When It Doesn't)
This method works when researchers have direct white-box access to the weights and hidden layer activations of a model. Ameisen uses it to verify whether specific internal concepts and circuits causally govern model behavior, rather than relying solely on superficial prompt outputs. If Claude translates between languages or shows an unexpected output bias, inspecting intermediate math reveals whether the model formed a stable concept or simply matched surface statistics.
The method breaks down when you run models behind commercial API endpoints. If you send calls to Claude or GPT-4 through a standard developer key, you receive text tokens, not intermediate layer tensors. You cannot inspect or edit intermediate vectors without hosting the weights on your own clusters. The process also demands massive storage and compute, because tracking billions of activation values across large datasets creates severe infrastructure overhead.
What to Do With This
If you build AI applications and Claude behaves erratically on specific customer inputs, stop guessing with prompt tweaks. Adapt Ameisen's four-step logic to your evaluation pipeline this week:
Step 1: Record Activations Across Arbitrary Data
Collect 200 real user transcripts, including successful completions and failures, and log the full message histories and tool call records for every run.
Step 2: Decode Concepts and Look for Statistical Patterns
Cluster your failure logs by semantic topic. Check if errors concentrate around specific concepts, like recurring invoice formats, ambiguous date strings, or currency symbols.
Step 3: Test Hypotheses with Targeted Prompts
Create synthetic test cases where you hold the prompt structure constant while varying only the isolated concept you identified. Verify whether that variable alone triggers the bad behavior.
Step 4: Causal Activation Steering
Intervene in the input pipeline before the model runs. Normalize or strip the triggering pattern in your pre-processing step, and measure whether your error rate drops across the benchmark.