Key Takeaways

  • Beren Millidge points out that poor sample efficiency forces labs into sim-to-real training, because models currently demand thousands of human interactions to master complex tasks.
  • Dwarkesh Patel highlights the core failure mode of modern deployments: live user interactions produce mountains of data, but almost none of it feeds back into base model weights.
  • Charlie O'Neill explains that Cursor bypassed purely synthetic environments by scoring real user telemetry with fast heuristics, followed by direct REINFORCE updates.
  • Economic limits will prevent continuous updates to giant base models; instead, real-world learning will happen through swapped-in LoRA adapters.

The Disagreement

Frontier AI labs are pouring compute into Reinforcement Learning with Verifiable Rewards (RLVR). The thesis is straightforward: build millions of synthetic environments inside data centers, let models run millions of trials, and transfer the resulting capabilities into the physical world.

Patel frames the scale of this bet: “The bet is that we will scale up RLVR training across millions of diverse environments, across hundreds of different kinds of domains.” Millidge agrees that this path is mandatory for now because “sim-to-real has to be the dominant framework while sample efficiency is low, because right now you need thousands and thousands of interactions with the humans.”

Yet running simulations in a server cluster hits a wall the moment an agent touches open-ended human work. Patel notes the absurdity of the current setup: real users generate billions of rich interactions daily, but “right now, that data is just not, in a meaningful sense, helping the model get better.”

O'Neill offers the counter-model: learn directly from live product exhaust. Instead of building massive synthetic worlds, Cursor looked at production logs. O'Neill explains: “Cursor’s fuzzy answer to this was, 'We have very good heuristics which are able to estimate how much better than average this response was, or how much worse than average this response was.' Then they would do this big REINFORCE update.”

Who's Right (and When They're Wrong)

Sim-to-real dominates when you have a fast, programmatic verifier. Compilers, formal proofs, and game engines provide unambiguous ground truth. If you can verify the output in milliseconds, synthetic RLVR in a cluster will beat human data collection every single time.

Sim-to-real fails on long-horizon, fuzzy tasks. You cannot simulate the social friction of an enterprise workflow or the subtle intent behind a developer undoing a refactor. When the reward signal is messy, synthetic environments train models to overfit on artificial proxies.

Updating a trillion-parameter base model on live user data is financially impossible. O'Neill identifies the real architecture: “I think the economics of this will pressure, not necessarily weight updates to one big common shared model, but modules that get subbed in. A very obvious example of this is a LoRA, but it might be something else.”

Synthetic training builds the general reasoning engine. Specialized, swappable adapters trained on real telemetry give that engine practical competence.

What to Do With This

Stop waiting for frontier model providers to solve domain adaptation for you. Open your product telemetry database tomorrow and identify three binary heuristics that indicate user satisfaction, such as an accepted diff, a zero-edit file save, or an interaction without a follow-up correction. Build an automated pipeline that bundles those positive and negative traces into nightly LoRA fine-tuning runs.