Key Takeaways

  • Autoregressive sequence models extract gradient signal across all tokens at once via parallel cross-entropy loss, making them far more training-data efficient.
  • Diffusion models sample noisy points along a continuous trajectory from data to noise, which dilutes the gradient signal across an infinite path.
  • At inference time, the trade-off reverses: autoregressive models suffer token-by-token sequential latency, while diffusion models can be distilled from 50 sampling steps down to 4.
  • Discrete diffusion models like Diffusion Gemma and Mercury use masking to jump between tokens, while continuous diffusion models handle representations in continuous vector space.
  • Black Forest Labs co-founders Andreas Blattmann and Dustin Podell argue that neither approach has won multimodal generation, which they are extending across image, video, audio, and robotics.

The Disagreement

Silicon Valley loves declaring a single architecture the winner. Large language models won text through autoregression, so many researchers assume the same next-token approach will run image, video, and action generation. The team behind FLUX 1 and Stable Diffusion sees a different reality.

Black Forest Labs co-founders Andreas Blattmann and Dustin Podell argue that the battle between diffusion and autoregression comes down to a hard split between training cost and runtime cost.

Autoregression crushes diffusion on training data efficiency. When you train a decoder model on a sequence, parallel cross-entropy loss computes loss over every single token simultaneously in one forward pass. Diffusion cannot do that.

As Blattmann explained: “I would say autogressive models are way more data efficient you can train them way better in data because every like you have for for diffusion models you have this trajectory from data to noise right and every training example you're training on is effectively because you have this continuous path from data to noise in between. It's giving you rise to infinitely many training examples.” (play from 1:14:18)

Because diffusion samples one random noise step along that continuous curve during a training pass, the model needs far more data exposures to learn the entire trajectory.

Inference flips the score entirely. Autoregression generates sequentially, one token after another. If your video generation model needs tens of thousands of tokens, you pay a steep latency tax on every frame. Diffusion, by contrast, generates the full output grid across continuous space.

Blattmann pointed out the operational edge: “However, at the sampling at sampling time or regressive models have to don't get across that. they have to autogressively create one token after the other because they're iterative in this like like data direction, right? And diffusion models you can just go down and distill 50 steps down to four and by that you can speed that up so much.” (play from 1:15:14)

Who's Right (and When They're Wrong)

Both sides are right, but in opposite environments.

Choose autoregression when you operate under severe data constraints or when exact sequential coherence dominates. Text reasoning and code synthesis demand discrete token logic. In those tasks, autoregressive transformers convert every piece of clean data into high-density gradient updates.

Choose diffusion when latency and throughput govern your product economics. If you run consumer media generation, real-time avatar interaction, or video synthesis, autoregression will break your server budget. A 50-step continuous flow matching model can be distilled to 4 steps or even 2 steps without destroying visual fidelity. Autoregressive token generation cannot skip tokens without degrading the output.

The technical line is also blurring. Podell noted the emergence of discrete diffusion models: “there are discrete diffusion language models which I almost don't I don't like to call them diffusion models really and this is where like the diffusion Gemma and like the mercury code like mercury models live which is is more of this like masking style model where where it it jumps from a mass to a token but it's still in the discrete space versus now there's a big wave of research right now going on continuous diffusion language models.” (play from 1:19:50)

Podell rejects the assumption that diffusion cannot handle broader reasoning tasks: “There's no inherent limitation to that you couldn't have a model that is diffusion based do the exact same process that can do all of these aspects.” (play from 1:16:38)

If your team is building visual intelligence for 2027 applications, such as factory robotics or real-time simulation, committing solely to next-token prediction leaves money on the table.

What to Do With This

Audit your generative inference stack this week. If you run autoregressive vision models in production, calculate your per-query cost and latency at scale: if your token count exceeds 1,000 per request, test a 4-step distilled diffusion pipeline like FLUX to benchmark the latency savings against generation quality.