Claire Vo: How to Audit 1,700 Pull Requests for Nine Cents

Key Takeaways

• Analyzing 1,700 pull requests across ChatPRD took two minutes and cost 9 cents using fast decision models instead of heavy vector embeddings.
• Running pairwise binary comparisons revealed 17,000 related pairs across the code repository without manual tagging.
• The resulting analysis showed that almost 30% of ChatPRD engineering effort went directly into platform security and infrastructure.
• The workflow pairs TypeSafe AI's decision model Jev for pairwise classification with Gemini Flash Lite for cluster labeling.
• Engineering leaders can run Claire Vo's Pairwise Repository Clustering Workflow to audit resource allocation across any git repository.

The Claire Vo's Pairwise Repository Clustering Workflow

• Step 1: Ingest Pull Request Metadata: Connect to GitHub via API and pull all pull request titles, descriptions, and metadata across the repository.
• Step 2: Run Jev Pairwise Binary Clustering: Use Jev to evaluate PR pairs (PR A vs. PR B) to determine whether they target the same thematic area or initiative using rapid boolean/choice checks.
• Step 3: Group into Thematic Clusters: Aggregate matching pairwise relationships into cohesive thematic clusters across the entire commit history.
• Step 4: Generate Cluster Labels via Low-Cost LLM: Pass the grouped clusters to a fast, cheap model (such as Gemini Flash Lite) to assign high-level business labels (e.g., platform security, reliability, documentation, prototyping).
• Step 5: Calculate Engineering Effort Distribution: Compute the percentage of total PRs allocated to each business initiative to report exact engineering effort breakdowns over time.

When This Works (and When It Doesn't)

This method works best on codebases with 100 to 10,000+ pull requests where engineering leadership needs visibility into where hours actually go without forcing developers to fill out Jira tags. When Vo tested the pipeline on a marketing site repository with 112 pull requests, the run cost 1.1 cents. Scaling to 1,700 pull requests cost nine cents.

The system breaks down when pull requests lack clear titles, descriptions, or scope. If your team merges single monolithic pull requests that bundle database migrations, UI redesigns, and bug fixes into one submission, pairwise comparison struggles to assign a single theme. The technique also measures pull request volume rather than lines of code or raw hours spent. A one-line hotfix carries the same initial weight as a multi-day architectural overhaul until you weight by commit depth or review cycles.

What to Do With This

If your board asks where your engineering bandwidth went last quarter, do not spend hours surveying engineers or guessing from sprint boards. Run this workflow across your primary repository this Friday.

First, generate a GitHub personal access token and pull the metadata for all pull requests merged in the last 90 days. Next, feed the titles and descriptions into a fast binary decision model to evaluate whether PR pairs share an initiative. Once your script groups the positive matches into clusters, send the clustered lists to Gemini Flash Lite with a prompt asking for a four-word business theme. You will walk into your Monday meeting with an exact breakdown showing what percentage of pull requests targeted bug fixes, customer requests, or internal tooling.