Key Takeaways

  • Post-training reinforcement learning produces very few actual information bits, making top capabilities trivial to distill once seen in public API outputs.
  • Chinese labs tap into proxy router services that bypass geographic blocks to capture live distributions of user prompts sent to US models.
  • Pre-training data provides little proprietary defense because frontier labs purchase data from the same commercial vendors open to global buyers.
  • Synthetic variation is cheap: Beren Millidge notes it is easier to direct a model to create a billion variants of an existing output than to create it from scratch.

The Distillation Trap

Frontier AI labs spend hundreds of millions of dollars training massive base models and running complex reinforcement learning pipelines. Yet within weeks of release, smaller open models often match their reasoning abilities. John Schulman explains why this happens: “Basically anything that can be learned through RL can be distilled very easily, because it’s a small number of bits.”

Reinforcement learning teaches a model how to search and format its reasoning, but it does not add vast libraries of new factual knowledge. The policy adjustments amount to a compact layer of behavioral updates. Once a model exposes those behaviors through an API, competing labs can simply query the model, record the chain-of-thought outputs, and train a smaller, cheaper architecture on those exact traces.

As Beren Millidge points out, generating data after the fact requires far less work than discovering it: "It’s actually much easier to say, "I want something like this," and then get the AI to produce a billion variations, than to actually create the thing like this to begin with."

How Router Services Leak Prompt Distributions

Distillation only works if you know what prompts real users actually send. If you distill on synthetic puzzles that nobody asks in production, your student model fails in the real world.

Competitors found a direct solution to this bottleneck. Schulman highlights a major leak in the distribution pipeline: “One thing that’s been coming out recently is that some of the Chinese companies are probably using these router services which are designed to allow people in China to use the US frontier models, which would otherwise be blocked in China.”

These third-party API proxies act as middle layers between users and Western providers. By sitting in the middle of that traffic flow, proxy operators log real-world user queries across coding, writing, and analysis. They pair those queries with the frontier model responses, creating the exact training dataset needed to replicate frontier performance without paying for original research.

At the same time, proprietary pre-training data advantages have eroded. As Millidge notes: “Just one other very basic point is that the frontier AI labs buy all their data from big data companies. The Chinese can also just buy the same data from data companies.”

The Trap of Benchmaxxing

To demonstrate progress, frontier labs often optimize for public evaluations. Schulman calls this the "benchmaxxing distribution," explaining that “it’s comparatively easy to create a lot of difficult environments that involve doing a much more complicated task or doing something that requires a lot more cleverness. You could say this is the benchmaxxing distribution, because a lot of the most prominent benchmarks just involve doing some very hard puzzle-like task that’s easy to verify.”

Because benchmark puzzles are easy to verify programmatically, labs pour compute into scoring high on math and coding tests. But fast benchmark gains do not equal uncopyable moats. When a capability is verified and released, competitors use the benchmark tasks and model outputs to replicate the exact same reasoning traces within months.

What to Do With This

Stop assuming your base model or fine-tune has a proprietary technological defense. If your startup's defensibility relies on reasoning techniques that a frontier lab can release next quarter, assume competitors will clone it in 60 days via distillation. Move your defensibility entirely to proprietary operational integrations, locked system access, and owned customer workflows that cannot be scraped by an API proxy.