Key Takeaways

  • Iterative fine-tuning on live user traces causes catastrophic forgetting after hundreds of micro-updates, erasing base capabilities.
  • Reinforcement learning trains general reasoning and skills well, but fails at absorbing granular, explicit domain knowledge like specific firm workflows.
  • Training a fresh base model from scratch with new data in mid-training consistently beats continually updating an existing model checkpoint.
  • Non-stationary data distributions cause continual weight training to hit an performance asymptote where the model stops absorbing new information.

The Illusion of Live Learning in Deployment

Every founder building AI workflows dreams of a system that learns directly on the job. You capture successful user traces, run supervised fine-tuning (SFT) or on-policy distillation on those traces, and expect the model to get smarter every afternoon. In practice, the weights break.

Charlie O'Neill points out the exact failure mode of this setup: “If I SFT the model on just successful traces, off-policy or on-policy, eventually in the very iterative regime, when you’re doing hundreds of these micro-updates, you see catastrophic forgetting.” When you update weights on small batches of fresh, non-stationary data, the optimizer overwrites older circuits that encode broad reasoning skills. The model gains a local trick at the cost of global intelligence.

This gap widens when teams try to force factual memory into model weights using reinforcement learning. As O'Neill explains: “RL is good at getting capabilities in, but it’s not as good at getting knowledge in, this very explicit knowledge of, 'Ah, okay, this person does this at this law firm, and this is a very specific process we find.'” Trying to make a neural network memorize specific client facts by tuning its parameters is using the wrong tool for the job.

Why Labs Retrain From Scratch Instead

When models hit deployment, incoming data changes constantly. User queries shift, domain requirements evolve, and edge cases pile up. If you continually feed this live stream into an existing model, the data distribution drifts away from the original pre-training mixture.

Beren Millidge describes what happens behind the scenes at frontier labs: “If you just naively train on non-stationary data, because you’re adding new data as you go, this is messing with the data distribution, so the old stuff is just forgotten.” The model stops retaining past skills because its plasticity is unconstrained across old representations.

Even worse, continuous weight updates hit a hard performance ceiling. Millidge notes: “If you just keep continually training the same base forever, it asymptotes at some point. You can’t just learn new stuff in that base. This is why people end up training new bases.” When you try to save compute by patching old weights forever, you end up with an inferior system. As Millidge states: “If you had some model with all this data, and you take literally the same-size model and pre-train it from scratch with all of the stuff in mid-training, it will be better.”

Until researchers invent new architectures with selective plasticity, continuous weight modification remains a dead end for production systems.

What to Do With This

Audit your pipeline this week. If you are fine-tuning base models on weekly batches of customer interactions, shut down that training run. Move customer-specific facts, firm rules, and dynamic context into retrieval-augmented generation and structured context windows, and restrict fine-tuning to stationary datasets designed purely for output formatting.