Key Takeaways

  • Current open-source video generation models, like 1.2.2, show a "night and day" quality gap compared to cutting-edge models like Kling or Veo, especially for long-form content.
  • Diffusion models hit a severe quadratic bottleneck for longer videos because their attention mechanism scales as N^2 (where N is sequence length), making 1-minute cinematic clips "almost impossible" with current compute.
  • Generating even 10-second sequences with full attention becomes computationally prohibitive, forcing compromises like sparse attention or frame stitching that degrade quality.
  • Ali Taha of Baseten argues that auto-regressive video models are the future for high-quality, long-form content, despite a current lack of good open-source options.
  • Unlike diffusion models' bidirectional attention, auto-regressive models process frames forward-only, allowing for LLM-like optimizations and token-by-token streaming crucial for extended sequences.

The Quadratic Wall: Why Diffusion Fails Long-Form Video

If you're building a product with generative AI, you know the gap between what's possible and what's practical. For video, that gap is a chasm. Ali Taha from Baseten pulls no punches: “If you look at the best video that you can generate today with an open-source model like 1.2.2 versus something like with Kling or Veo, the difference is night and day.” The problem isn't just a lack of polish, it's a fundamental architectural limitation.

The core issue for current diffusion models, which dominate video generation today, is how they handle long sequences. Their attention mechanism scales quadratically. As Taha explains, “The attention becomes such a huge bottleneck. And because it's over N square, if you're doing like if you extend that to like 10 seconds, well, it's just N square. 20 seconds, 30 seconds. So, to generate a good cutscene of like 1 minute, it's almost impossible to do within the same compute time.” Imagine trying to render a cinematic shot that lasts more than a few seconds. The compute required for a full-attention diffusion model explodes, making it economically and technically unfeasible.

The Auto-Regressive Bet for Cinematic Content

Facing this quadratic wall, developers are forced into compromises. You can bite the bullet and throw enormous compute at the problem for full attention over millions of tokens, hoping to generate a 2-minute video. Or, as Taha suggests, you look to a different paradigm: “Either you bite the bullet, you have huge compute, and you do full attention over like a million tokens because you're trying to generate like 2 minutes of video. Or you move towards auto regressive video.” He makes it clear: “Auto regressive video seems to me like that is the bet that the future is going to be making.”

Philip Kiely highlighted the key difference: “The key thing to understand between a auto-aggressive model and a diffusion model is that diffusion attention goes in both directions while auto-aggression it only goes forward in the sequence.” This forward-only processing is a game-changer for long-form content. While today's auto-regressive video models lag in quality and consistency, their architecture allows for LLM-like optimizations, including token-by-token streaming. This means they don't need to hold the entire sequence in memory to make sense of it, sidestepping diffusion's quadratic bottleneck and opening the door for truly long, consistent, and high-quality video generation.

What to Do With This

If you're building in the generative AI space, especially with a video component, stop planning on diffusion models for anything longer than short clips. Start tracking the nascent auto-regressive video scene and consider what talent with deep LLM architecture experience could bring to your team. The next leap in video generation won't come from brute-forcing compute on old architectures; it will come from a strategic pivot to models built for scale and sequence.