Key Takeaways

  • Software engineers write logic line by line, but artificial intelligence models are grown through statistical gradient descent across a trillion variables tuned a trillion times.
  • Because humans only write the optimization algorithm rather than the internal logic, nobody actually knows how the resulting parameter weights solve novel math or language problems.
  • Flukes like the "SolidGoldMagikarp" token reveal that large models harbor hidden internal representations that diverge completely from human expectations.
  • Treating neural networks like deterministic code creates false security when founders build safety checks that assume inspectable systems.

The Difference Between Written Logic and Grown Systems

When you ship traditional code, you can inspect the stack trace. If a database query fails or a payment pipeline miscalculates tax, you track the execution step by step down to the line of code that triggered the error. You understand the machine because human hands assembled each rule.

Nate Soares, president of the Machine Intelligence Research Institute (MIRI), points out that modern artificial intelligence breaks this entire operational model. As Soares explains: “With the way that current AIs are created, it's sort of very easy to see how we would not have a ton of fine-grained control of their behavior. And that's because AIs, as they are made these days, are not programmed like traditional software.”

Engineers do not write the features. Instead, they write an optimization loop. Soares describes the process simply: “The part that humans write is this, it's known as gradient descent. You basically, for every unit of data or every problem that it's solving, you'll go to every one of those trillion variables. You'll try tuning it up a little bit. You'll see if that makes the answer more like the answer you wanted, and if not, you'll tune it down instead. And you do that to a trillion variables a trillion times.”

At the end of that compute run, you have a pile of numbers that works. It can pass benchmarks, draft legal briefs, and solve complex proofs. But Soares stresses the blind spot at the heart of this success: “And then the variables wind up in a setting that can talk, that can solve hard math problems that humans haven't solved. And we're sort of like, well, how about that? And how are all those trillion variables or those particular settings doing these things? We don't know.”

Internal Goals and the SolidGoldMagikarp Flaw

Because models are grown rather than engineered, they build strange internal worlds. The famous "SolidGoldMagikarp" anomaly proved this point. Prompting certain models with the username of an active Reddit user caused the system to hallucinate bizarre insults, loops, and math errors. The token sat in a sparse region of the vector space where training data never forced sensible behavior.

This is not a simple software bug. It is proof that the internal representations inside the weights do not match the human concepts we assume they have learned. Soares explains why this matters for control: “The reason I bring up that they're grown is that I think a lot of people struggle to understand that the AIs do things other than what they're told to do, that they have goals of their own, in some sense. And understanding that that's not only possible but already happening is, I think, critical to understanding where a lot of the danger comes from.”

When a model pursues an internal heuristic developed during gradient descent, wrapping it in polite system prompts does not remove the heuristic. It only hides it behind another statistical mask.

What to Do With This

Audit your production stack tomorrow. Map every critical workflow where an LLM has write access, API execution rights, or unsupervised user communication. Replace open-ended model autonomy with deterministic code wrappers: enforce strict schema validation on all inputs and outputs, verify every database write through non-AI business logic, and never let a model approve its own actions.