Key Takeaways

  • Deploying an AI agent onto enterprise cloud infrastructure does not make it production-ready. Without structured evaluation benchmarks, Kevin Solorio argues that any deployed system remains an unverified proof of concept.
  • Blue-collar operations lack the neat digital paper trails of software desks. To extract real operational knowledge, builders must present frontline experts with flawed synthetic drafts rather than asking open-ended interview questions.
  • A live evaluation loop requires two engines: deterministic unit tests for rigid code logic and LLM-as-judge models to score open-ended production traces against domain rubrics.
  • Unresolved edge cases and execution failures cannot sit in unread logs; routing them to a dedicated annotation queue converts production errors back into offline test fixtures.
  • Teams build reliable field tools by running Solorio's Continuous Agent Evaluation Flywheel to link real-time user corrections directly to test suites.

Solorio's Continuous Agent Evaluation Flywheel

At the Midwest AI Summit, Solorio challenged the obsession with deployment plumbing. “An agent without evals is a proof of concept,” Solorio said. “And this doesn't matter what infrastructure it's been deployed onto.” When building for specialized industries, you cannot wait for perfect data. You have to manufacture the initial interaction, capture domain knowledge on the fly, and harden your evaluation baseline over time.

Step 1: Synthetic Seeding and Deliberate Flaws

Synthesize an initial test dataset to get an interactive prototype into users' hands quickly. Intentionally display outputs (even flawed ones) because domain practitioners react much faster to correct tangible errors than to answer abstract requirement questions.

Step 2: Low-Friction Labeling Interfaces

Provide simple interfaces (such as Google Spreadsheets or lightweight gamified apps) so field representatives can review outputs and label corrections asynchronously without friction.

Step 3: Offline Benchmark Development

Consolidate vetted feedback into an offline evaluation dataset used as the testing baseline to measure agent iterations and prevent regressions during development.

Step 4: Online Trace Evaluation

Deploy deterministic code evaluators and LLM-as-judge scoring models to continuously sample and evaluate live production traces.

Step 5: Annotation Queue and Feedback Loop

Route unhandled or failed traces into an annotation queue for human review. Once diagnosed, append these real-world edge cases back into the offline evaluation suite to close the loop.

When This Works (and When It Doesn't)

Solorio designed this framework for specialized, physical industries where clean training logs do not exist. When you build for HVAC technicians, warehouse dispatchers, or factory maintenance crews, the operating logic lives inside workers' heads. Solorio noted that getting requirements from busy people is painful, but “if you show them something that's wrong, they are very quick to tell you where it is incorrect.” Giving domain experts an incorrect draft unlocks instant feedback.

This method breaks down in safety-critical environments where bad outputs cause direct physical danger or legal liability. If a field technician acts on a deliberately flawed agent recommendation while servicing a commercial electrical panel, you create severe risk. In regulated, high-hazard settings, isolate the deliberately flawed outputs within sandboxed review environments rather than active operational tools. It also fails if frontline workers view labeling as unpaid overhead. If your feedback interface takes more than two clicks, workers will bypass it entirely.

What to Do With This

If you are building an AI dispatch assistant for field technicians this week, stop setting up forty-minute discovery interviews. Your experts will give you vague platitudes because their real judgment is instinctual.

Instead, generate twenty synthetic dispatch schedules containing deliberate, realistic mistakes: assign a junior apprentice to a complex commercial repair or route two jobs across town during rush hour. Paste those schedules into a shared Google Sheet with three simple columns: Suggested Assignment, Correction, and Why.

Send the sheet to your lead dispatcher with a note asking them to fix what you broke. By day two, take their corrections and write your first deterministic test assertions against those specific errors. Before pushing any new model prompts next week, run your test runner against this offline suite. When a production dispatch gets manually overridden in the field, pipe that raw trace into an Airtable triage queue, fix the root cause, and append it as test case twenty-one.