Why App Shots Beat Raw Screenshots for AI Agents
Ari Weinstein explains why App Shots use accessibility trees and DOM metadata instead of raw pixels to give OpenAI models reliable computer control.
OpenAI’s New Agent Stack: Computer Use, Decisions API, UltraFast, Dots—Ari Weinstein & Nikunj Handa
With swyx, Ari Weinstein, Nikunj Handa · Sunday, October 4, 2026
Recorded immediately following OpenAI DevDay, Ari Weinstein and Nikunj Handa join the hosts to detail OpenAI's newest agent and API announcements. Weinstein explains how Computer Use evolved to become faster than average human users through multimodal representations, code-driven multi-step execution, and cloud Linux VMs in Dots. Handa outlines the developer API stack updates, including the Decisions API, parallel batching, WebSockets, UltraFast inference, compaction techniques, and extended prompt caching.
Key takeaways
- Raw screenshots hide critical structured data from language models, including hyperlink URLs and truncated calendar text. Read more →
- OpenAI introduced 12-hour prompt cache guarantees to slash costs for persistent agent loops that outlive standard 30-minute eviction windows. Read more →
- OpenAI built the Decisions API in under four weeks without training a single new model weight. Read more →
- OpenAI provisions each Dots assistant with its own dedicated Linux virtual machine in the cloud, bypassing the limits of simple browser scraping. Read more →
- Synchronous tool calls create artificial latency bottlenecks; async function calling lets the model reason and generate continuously while external operations execute. Read more →
- Early Computer Use systems trapped agents in slow loops: take a screenshot, scroll down, take another screenshot, and guess the next pixel coordinate. Read more →
6 articles from this episode
OpenAI's Strategy for Long-Running Agent Contexts
OpenAI's Nikunj Handa reveals 12-hour prompt cache guarantees, pre-warming APIs, and server-side context compaction for agents.
Inside OpenAI's Decisions API: Sub-Second Classification
How Nikunj Handa and OpenAI built the ultra-fast Decisions API in under four weeks using existing Luna weights and parallel question execution.
Why OpenAI Built Cloud Linux VMs for Dots Assistants
Ari Weinstein explains why OpenAI gave Dots persistent cloud Linux virtual machines to run desktop software and automate workflows.
How OpenAI Built Mid-Turn Steering and Async Tool Calls
Nikunj Handa details OpenAI's move to WebSockets, async tool execution, and mid-turn steering for real-time agent architectures.
Why OpenAI Abandoned Screenshot Loops for Code Execution
Ari Weinstein explains how OpenAI Computer Use evolved past visual loops to batch actions via DOM inspection and JavaScript.
More Latent Space episodes
- The Future of Claude Code: Mods, Mutable Software, & Multiplayer Agents — Thariq Shihipar, AnthropicOct 4 · 4 articles
- 🔬 Google's AI Scientist Started as an Attempt to Automate Kaggle — John Platt, Google FellowSep 27 · 5 articles
- 🔬Bio-security is an AI Arms Race - Eric Nguyen (CEO, Radical Numerics)Sep 27 · 3 articles
- The $10 Trillion Token Economy — Alex Atallah, OpenRouter & Anjney Midha, AMPSep 27 · 5 articles