Key Takeaways

  • Static coding benchmarks fail on consumer agents because real user behavior spans multi-turn, multi-day interactions rather than single prompts.
  • Mona Sarantakos deployed autonomous agent swarms at Meta Superintelligence Labs to inspect live user conversations and generate synthetic edge-case data.
  • The engineering team turned "swarm it" into their standard triage routine, using automated swarms to confirm whether anomalies were real product bugs before assigning developers.
  • The swarms produced structured reports called “dumps from the swarms” (play from 38:57) that surfaced user tasks and qualitative friction points instead of raw numerical metrics.

The Method

When Mona Sarantakos and her team began building Muse, Meta's personal AI agent, existing model evaluation suites proved useless. Traditional benchmarks measure single prompts, math puzzles, or isolated coding tasks. Muse had to handle consumer requests that extended across multiple turns and days. As Sarantakos explained: “And then when we started building this, it was like completely multi-turn, right? Like it was like back and forth, back and forth, back and forth. And we don't have data that looks like that.” (play from 39:46)

Static metric dashboards showed completion rates and latency, but those figures concealed why users dropped off midway through a complex task. To fix this blind spot, the engineering team deployed autonomous agent swarms across production interactions.

The swarms fulfilled two distinct functions:

First, they synthesized qualitative patterns into structured briefs called "dumps from the swarms." Rather than drowning engineers in raw transcript logs, the swarms grouped conversations into formatted overviews of user goals, edge-case breakdowns, and friction points.

Second, the swarms generated synthetic multi-turn data to fill training and evaluation gaps. Because the team had no historical corpus of multi-day consumer agent interactions, synthetic conversations allowed them to stress-test the model against back-and-forth dialogue before shipping changes.

This workflow transformed bug triage into a rapid verification process. When an unexpected edge case surfaced, the team did not immediately assign a ticket. As Sarantakos put it: “The engineers were like oh we'll swarm it and figure it out. Swarmed it, and like, oh no, this is an actual problem that we need to solve, and we're going to get somebody on it.” (play from 40:46)

Looking at qualitative summaries instead of pure metrics changed how the team operated. Sarantakos noted that because the swarms brought back insights rather than just numbers, it created direct empathy for user struggles, helping the team feel the exact pain users experienced.

Where This Breaks Down

Autonomous swarms work when an agent handles high conversational volume across open-ended workflows. If your product has fifty users, do not build an evaluation swarm. Automated synthesis filters out human nuance; at an early stage, founders learn more by reading every transcript manually.

Secondary agent swarms also create a cost tax. Running evaluation models across every production token multiplies inference bills quickly, which makes continuous swarm evaluation unsustainable unless tightly scoped to specific sample sets.

Finally, synthetic multi-turn data carries risk. If your synthetic agents evaluate conversational flows based on the same model assumptions powering the core agent, you create an echo chamber that blinds the team to bizarre, non-linear human inputs.

What to Do With This

Pull your ten most abandoned multi-turn conversations from the past seven days. Prompt an LLM to identify the exact turn where conversation stalled and output a one-line summary of the user's unmet intent. Review those ten summaries with your lead engineer tomorrow morning to decide if the drop-off stems from a model failure or a product design flaw.