Key Takeaways

  • Mid-training on synthetic reasoning datasets does roughly 80 percent of the work before reinforcement learning begins.
  • A single binary outcome bit in RL delivers a cleaner learning signal than noisy token-by-token imitation across long thought chains.
  • Small parameter updates from sparse outcome rewards create massive changes in the model's actual function space.
  • Reinforcement learning teaches models time-horizon generalization, training them to spend more tokens productively on hard problems.
  • John Schulman cautions that aggressive RL creates stylistic monoculture and entropy collapse, forcing models into repetitive tropes.

The One-Bit Signal and the Mid-Training Secret

Most people assume reinforcement learning teaches a base language model how to reason from scratch. Beren Millidge argues that assumption is backwards. The real foundation gets built earlier during mid-training, where researchers run pre-training objectives on dense, synthetic reasoning traces.

“First, what is slightly underestimated is the mid-training,” Millidge explained. “An awful lot of what we see as successes of RL actually comes from very, very good mid-training data, which is where we’re essentially doing pre-training but on synthetic reasoning data and the kind of environments that get the model warm-started for RL.”

Once that warm-start happens, RL steps in with a distinct mathematical advantage: sparsity. In standard supervised fine-tuning, every single token gets forced toward a target. If the model writes a 2,000-token proof, supervised learning grades all 2,000 tokens, introducing massive noise from irrelevant phrasing choices. RL discards token-level supervision and scores only the final answer.

“For RL, you only get the one bit,” Millidge said. “That means that signal is not drowned out in the noise of all the other bits the model has.” Even though one bit sounds tiny compared to billions of parameters, its effect is outsized: “Even if you have a few bits and you only change the parameters a small amount, the actual impact on function space (the input-to-output mapping the model learns) can still be super dramatic.”

Long Horizons versus Style Collapse

When you give a model binary rewards for correct answers on difficult problems, its behavior shifts. Charlie O'Neill pointed out that RL primarily teaches the model how to stretch its compute over time without going off the rails.

“The models just learned how to use more tokens for longer and still make progress on some sort of task,” O'Neill noted. The model learns internal search, backtracking, and self-correction simply because longer deliberation paths increase its odds of scoring that final positive bit.

Yet this optimization has a dark side. As the model narrows its focus strictly to high-reward trajectories, it loses diversity. John Schulman warned that this leads to severe entropy collapse and behavioral narrowing.

“Even though the models seem like they’re good at writing, when you do some kind of distributional analysis, you find that they’re reusing certain themes all the time and they’re using the same character names all the time,” Schulman observed. When you train purely on outcome verification, the model finds a narrow ridge of safety and stays there.

What to Do With This

Stop trying to fine-tune your reasoning agents with standard supervised input-output pairs. If you are building automated evaluation or specialized reasoning pipelines this week, generate verified synthetic solution traces for initial adaptation, then apply binary pass-fail verifiers on the final output instead of scoring intermediate syntax.