Key Takeaways

  • Large language models fail at proof-checking because probabilistic text generators cannot guarantee logical consistency.
  • Modern automated mathematics couples an LLM with Lean, a programming language built on deterministic logical axioms.
  • Instead of outputting mathematical English, frontier models produce machine-readable Lean code that compiles directly against a library of verified facts.
  • AI-generated proofs now stretch to hundreds of thousands of lines of code, making manual human verification impossible.

The Flaw in Generative Reasoning

If you ask an AI model to solve a novel math problem, it will write an answer that reads like textbook mathematics. It will use standard symbols, cite real theorems, and adopt an authoritative tone. Often, the argument falls apart halfway through. Probabilistic models guess the next token based on statistical patterns. Mathematics requires strict deduction where a single logical gap destroys the entire structure.

Justin Solomon, associate dean of engineering education at MIT, points out that AI models are surprisingly bad at catching their own errors. As Solomon explains: “And in fact, actually they're quite bad at proof-checking. And one of the revolutions in, I would say, the last year or so is the realization that actually what you have to do is to provide your model, like your large language model, with a second module that can check the AI's proof.”

Trying to fix hallucination by prompting a model to review its own logic leads to circular reasoning. The model simply validates its own plausible-sounding mistakes.

Offloading Truth to Deterministic Code

To solve this bottleneck, researchers stopped asking models to verify proofs in plain English. Instead, they paired probabilistic neural networks with deterministic interactive theorem provers. The primary tool for this work is Lean.

Lean is not an AI system. It is a formal programming language based on a minimal set of foundational axioms. Solomon explains how it works: “So this programming language, Lean, what it does is it has a few simple, maybe, axioms that are built in, so math things that we sort of agree are the starting points for writing proofs, like the steps of basic logic.”

When a model attacks a conjecture today, it outputs executable Lean code instead of prose. The Lean environment checks every deduction step against its axiomatic rules. If a step follows the rules, Lean accepts it and stores the result in its library of validated facts. If a step breaks logic, the compiler throws an error and rejects the line. The model receives that rejection as immediate feedback, adjusts its attempt, and tries another path.

This separation of roles solves the reliability problem. The LLM acts as an intuitive explorer, searching for paths and suggesting candidate ideas. Lean acts as an unforgiving compiler that verifies every step before admitting it to the record. Solomon notes: “So actually the correctness part of it is usually deferred to another piece of software which isn't AI, but somehow, because of the interaction between the AI tool and this proof checker, that's what's created a lot of progress that we can believe in a little bit more than we did in the past.”

The Rise of Unreadable Verification

This division of labor changes what mathematical work looks like. In the past, a mathematician wrote a paper for other humans to read and critique over months or years. Today, AI models generate proofs that skip human readability entirely.

“Some of the modern proofs that you're seeing, especially the AI-generated ones, are like hundreds of thousands of lines of Lean code,” Solomon says. “I don't think a human could check it.”

Nobody sits down to read a 200,000-line Lean file line by line. You do not check the proof; you run the compiler. If the compiler executes without errors from axiom to conclusion, the proof is sound. Human review shifts from validating individual inferences to auditing the original formal definitions and trusting the compiler kernel.

What to Do With This

Audit your product pipeline for places where you ask an LLM to evaluate its own output. If your system relies on an AI judge to confirm that an AI generator got an answer right, tear out that evaluation loop. Replace it this week with a deterministic validator: a strict schema parser, a unit test suite, or a deterministic rules engine that rejects bad outputs without human or model intervention.