Key Takeaways

  • The How I AI benchmark evaluates frontier models on roughly 20 practical workflows divided into five buckets: prototypes, agentic tasks, coding, writing, and creative output.
  • Evaluation combines human judgment with automation: Claire Vo records rapid voice notes into Claude, which merges her ratings with an LLM judge to build final slides.
  • Meta gates access to its Muse Wonder model behind Muse Code, a separate terminal tool that interrupts existing engineering environments.
  • Rather than adopting Meta's standalone interface, Vo granted Claude Code direct access to Muse Code so it could run headlessly in the background.

The Method

Claire Vo built the How I AI benchmark to test models against daily engineering and product demands rather than academic puzzles. As Vo explains, “we run against almost 20 different tasks. That's everything from writing a PRD to doing a set of prototypes to redesigning a chat PRD site to running long-running computer use, agentic research, code repair tasks, and then some creative tasks.”

To make that spread usable, she separates tests into five distinct groups: “prototype, agentic work, coding work, writing work, and creative work.”

Her scoring process removes the drag of filling out spreadsheets across twenty complex evaluations. Instead of typing manual rubrics or relying solely on automated scripts, she feeds qualitative data straight into her model. “Use voice notes to Claude,” Vo says. “Claude takes my scores, the LLM as a judge ranks them, and then puts them together in a presentation, which he's working on right now.” Speaking reactions while looking at UI designs or reviewing pull requests captures nuanced product instincts, while Claude normalizes the numbers and handles presentation formatting.

The real operational bottleneck appeared when accessing Meta's Muse Wonder model. Meta required testers to install a dedicated client. “Okay, now let's talk about the most annoying part of Muse Wonder,” Vo notes. “It is that you have to download Muse Code.”

Instead of abandoning her core terminal setup or learning an unfamiliar interface, Vo automated the interaction entirely. She avoided running the client manually: “I signed up. I gave Claude code access to Muse code and ran it headless-ly.” Claude Code operated as the primary driver, executing commands through Muse Code in the background and returning outputs directly to her preferred workspace.

Where This Breaks Down

Headless agent nesting creates hidden debugging problems. When you instruct Claude Code to pilot Muse Code, you place two nondeterministic layers back-to-back. If Muse Code fails, drops terminal context, or times out, Claude Code has to guess whether the failure came from the underlying model or the wrapper executing it. Diagnosing errors across stacked agents takes twice as long as diagnosing direct CLI output.

Subjective scoring via voice notes also introduces mood variance. While the LLM judge provides a baseline score, human voice recordings capture momentary fatigue. If you evaluate five coding tasks at midnight after fighting broken package dependencies, the voice feedback Claude processes will skew harsher than a morning run on fresh eyes. Keeping the benchmark valid requires strict scoring rubrics before you record.

What to Do With This

Audit your internal model testing setup this week. Select three practical workflows, such as writing a feature specification or fixing failed unit tests, and run them through your primary coding agent. If an AI provider requires a proprietary terminal client, do not change your daily habits: grant your existing CLI tool headless execution rights to call the secondary binary and return the outputs directly.