Key Takeaways

  • NVIDIA research called AVO proved that wrapping a model in a dedicated reasoning and tool system lifted benchmark accuracy on ARC-AGI from 30% to 100%.
  • Querying closed frontier labs for continuous security telemetry is financially impossible for enterprise fleets operating 24 hours a day.
  • Open base models fine-tuned on private threat data beat raw frontier models on domain accuracy and compute costs.
  • Justin Boitano frames frontier models as battleships, while effective enterprise defense requires cheap, agile drone models running across millions of endpoints.

The System Around the Model Wins the Benchmark

Raw model weights get all the press, but engineering the scaffolding around them delivers the actual performance leap. At CrowdStrike's Fal.Con conference, NVIDIA executive Justin Boitano pointed to internal research on an agent system called AVO. The team took a frontier model scoring 30% on ARC-AGI benchmarks and pushed it to 100% accuracy simply by changing the tool routing, error correction, and execution loops wrapped around the model.

“It's not really about just the model, it's about the entire system,” Boitano explained.

When founders chase marginally better foundation models from OpenAI or Anthropic, they burn cash on compute while leaving execution quality to chance. A smaller model wrapped in strict verification loops, schema validation, and domain-specific tools consistently outperforms an unconstrained frontier model in production tasks.

The Battleship Problem and the Economics of 24/7 Monitoring

Defending an enterprise requires continuous observation across hundreds of thousands of endpoints. Routing that entire data stream through closed API endpoints creates an unsustainable cloud bill.

“We've heard it loud and clear from customers that just going one direction with Frontier Labs is just too cost prohibitive,” said Daniel Bernard, CrowdStrike's Chief Business Officer.

Boitano compared general frontier models to battleships: massive, expensive, and slow to deploy everywhere. Security teams do not need a single billion-dollar dreadnought sitting offshore. They need fleets of cheap, distributed drone models running continuously on local hardware across the entire estate.

When you run millions of automated checks per minute, latency and token pricing dictate system architecture. Raw intelligence is cheapening rapidly, but running heavy frontier calls on routine log analysis remains an economic dead end.

Domain Telemetry Turns Spinning Compasses Into Weapons

Off-the-shelf open models are not enough on their own. As Bernard put it: “Generic open source at this point is sort of like a compass that is spinning in a circle.”

The advantage belongs to teams that feed open weights with proprietary telemetry. CrowdStrike takes accessible base models and trains them on real attack patterns collected across their customer base. That specialized tuning transforms an unfocused base model into a sharp defensive system.

“Intelligence, I believe, is really becoming somewhat commoditized,” Bernard noted. “What is really real in this next chapter of AI is how you contextualize based off of specific situations.”

The moat is not owning an enormous foundation model. The moat is having proprietary telemetry, structuring tight agent workflows, and operating at a cost structure your competitors cannot match.

What to Do With This

Audit your AI features this week and identify your top token cost center. If you are sending routine data checks to GPT-4o or Claude 3.5 Sonnet, replace that single endpoint with a smaller open model wrapped in deterministic schema validation and multi-step tool loops. Measure the task accuracy and cost delta over 100 test runs before committing more budget to frontier API credits.