Databricks' Dream Engine: Build the Optimization Factory First
Databricks rewrote its database engine with AI, learning from quadrillions of data points. Founders, stop optimizing your systems. Build a system that optimizes itself.
The Agent Cloud: Databricks’ Bet on the Future of AI — Matei Zaharia and Reynold Xin
With swyx, Matei Zaharia, Reynold Xin · Sunday, June 28, 2026
This episode features Databricks co-founders Matei Zaharia and Reynold Xin discussing their latest innovations, including the Omnigents platform for agent development and L-TAP, a novel approach to unifying transactional and analytical databases. They delve into Databricks' core strategies around open-source, AI integration, and a unique 'Dream Engine' initiative, while also drawing comparisons with competitors like Snowflake and clarifying their post-MosaicML LLM strategy.
Key takeaways
- Databricks co-founder Reynold Xin revealed the "Dream Engine," an ambitious project to rewrite their database engine from scratch, directly addressing the limitations of existing decade-old designs. Read more →
- Databricks, even after releasing DBRX, is explicitly moving away from competing in the race to build general frontier LLMs. They are focusing on specialized models and systems. Read more →
- Databricks' Omnigents is an open-source Agent Cloud platform, aiming to standardize how AI agents are built and collaborate. It provides a common API across different agent environments, like various harnesses and cloud sandboxes. Read more →
- Databricks started with an unwavering commitment to open data formats like Parquet and Delta Lake, a counter-intuitive bet that paid off as enterprises rejected vendor lock-in. Snowflake initially prioritized proprietary formats. Read more →
4 articles from this episode
Databricks: Why General LLMs Are a Trap. Specialize Instead.
Matei Zaharia reveals Databricks' post-MosaicML LLM strategy: ditch general AGI quests for cheaper, more accurate specialized models. Founders, focus on fine-tuning.
Databricks' Omnigents: The $5 Cap on AI Agent Overspend
Databricks co-founder Matei Zaharia explains Omnigents, their open-source agent cloud, focusing on contextual policies that cap agent spending and data access.
Databricks Founders: Open Data, AI-First Beat Snowflake's Strategy
Databricks co-founders Matei Zaharia and Reynold Xin explain how an unwavering commitment to open data formats and early AI focus outpaced Snowflake.
More Latent Space episodes
- The Future of Claude Code: Mods, Mutable Software, & Multiplayer Agents — Thariq Shihipar, AnthropicOct 4 · 4 articles
- Which GPU Clouds Are Actually Good? | ClusterMAX 3.0Oct 4 · 6 articles
- Recursive Language Models — Alex Zhang, MIT PhDOct 4 · 6 articles
- OpenAI’s New Agent Stack: Computer Use, Decisions API, UltraFast, Dots—Ari Weinstein & Nikunj HandaOct 4 · 6 articles