Key Takeaways

  • Engineering teams waste weeks building thousand-question benchmarks before they understand what good output actually looks like.
  • Reliable evals start with 5 to 20 hand-crafted prompts on tasks where you have personal taste and current models show partial success.
  • Automated judge models can distort results if you do not test them alongside your full production filters and mitigations.
  • Google had to upgrade its image evaluation from the old six-tone Fitzpatrick scale to the 10-tone Monk Skin Tone scale to detect real failures in visual models.
  • Founders should run Doshi's First AI Eval Creation Method before writing automated grading pipelines or hiring labeling firms.

Doshi's First AI Eval Creation Method

Step 1: Pick a High-Stakes Domain You Care About

Select a task or use case you are personally invested in and have deep taste in (e.g., strategic PRD writing, complex financial modeling, or visual generation) where current models show promise but are not yet perfect.

Step 2: Curate 5 to 20 Stress-Test Prompts

Start simple instead of building a massive benchmark. Write 5 to 20 focused prompts designed to push the model's capabilities and reveal differential signal between Model A and Model B.

Step 3: Manually Act as the Human Rater

Review the multi-turn trajectories and outputs yourself. Define what 'good' actually means by calibrating your own subjective standards against the responses.

Step 4: Formalize Rubrics and System Tests

Turn your observations into clear evaluation rubrics, ensure automated judge models are rigorously calibrated, and test the outputs within the full end-to-end serving and filtering stack before scaling.

When This Works (and When It Doesn't)

Doshi's approach works when an engineer or product manager needs to establish a fast evaluation baseline. If you are deciding whether to swap your core model from Claude 3.5 Sonnet to Gemini 1.5 Pro, running twenty sharp prompts exposes behavioral shifts in an afternoon. Massive benchmarks like MMLU hide the specific failure modes that irritate your actual users.

This method stumbles once you need regression testing across rare edge cases. A human rater reviewing twenty outputs cannot catch a 1% safety failure or subtle data drift across thousands of daily requests. When Google updated visual fairness testing, moving from the six-tone Fitzpatrick scale to the 10-tone Monk Skin Tone scale, human intuition alone was insufficient. They needed structured, objective buckets across production traffic. Start with your personal taste to discover the rubric, but transition to automated suites once you deploy to paying customers.

What to Do With This

Suppose you are building an AI tool that converts messy customer support transcripts into structured product bug tickets. Do not open an evaluation software platform or generate 500 synthetic examples this week.

First, pick five actual transcripts from your company history that confused junior support reps. Next, run those five transcripts through both your existing prompt and your proposed model update. Sit down with a blank text document and read the multi-turn outputs yourself. Mark exactly where the model hallucinated user intent or missed reproduction steps. Turn those notes into a five-point grading checklist. You now have an eval that reflects real product taste, built in ninety minutes.