Key Takeaways

  • Handshake AI built VIALS to test frontier vision models on wet lab tasks like Western blot gels, plasmid maps, and histological tissue slides.
  • The best multimodal models score between sub-20% and 50% accuracy on basic laboratory visual tasks that a high school intern can perform.
  • Verbalized model confidence is broken: systems report near-100% certainty on completely incorrect diagnostics, including misidentifying necrotic brain tissue.
  • Nick Magazine warns that commercial lab automation requires sustained accuracy above 90% because lower reliability creates unacceptable economic and health liabilities.

The Multimodal Biology Blindspot

Silicon Valley pitches AI as an engine that will cure every human disease within a decade. The reality inside wet labs tells a different story. Frontier multimodal models fall apart when asked to interpret the standard visual assays that junior technicians read every afternoon.

Nick Magazine and Jonas Mueller at Handshake AI built VIALS to measure this specific gap. The benchmark tests vision-language models on the core visual artifacts of modern life sciences: Western blot gels, plasmid maps, and histological slices. The results expose a wide ditch between conversational fluency and functional perception.

“Most models out of the box do fairly poorly,” (play from 33:23) Magazine explained. “All of them perform at 50% or worse. Even the very top of the frontier models, which if you're a scientist is highly problematic.” (play from 33:27)

On multiple tests, accuracy collapsed below 20%. Models fail to resolve ambiguous visual bands on standard gels, misread plasmid vectors, and misclassify sections of necrotic brain tissue. As Mueller put it: “AI is not going to cure cancer before it can interpret these very basic types of images.” (play from 0:06)

Blind Certainty in High-Stakes Environments

Low accuracy is manageable if a model knows when it is uncertain. A system that scores 45% can still route hard images to a human pathologist. The actual failure mode in VIALS is worse: the models lie about their own competence.

Mueller and his team tested verbalized confidence by asking models to self-report how sure they were of each answer. The calibration was inverted. “Totally vibes based based on the model's intuition,” (play from 1:01:51) Mueller said. “We find they're notoriously uncalibrated, and they often say they're at max confidence, basically near 100, and the answer is wrong.” (play from 1:01:54)

In an academic benchmark, false certainty ruins a leaderboard score. In a wet lab, it ruins cell lines, invalidates clinical research batches, and risks human health. Magazine pointed out that commercial deployment requires a standard far above coin-flip territory: “Somewhere under 90% would be more or less inserviceable. I mean there's a real economic and also in some cases public health impact of getting this kind of thing wrong.” (play from 59:53)

If you drop today's frontier vision models into a closed-loop biological workflow, they will misread your gel, hallucinate the target band, confirm that their diagnosis is 100% correct, and instruct the liquid handler to waste thousands of dollars of reagents on the wrong clone.

What to Do With This

Take 50 raw image outputs from your primary operational pipeline this week, whether they are Western blots, schematics, or invoice scans. Run them through your current multimodal model, prompt it to state its confidence from 1 to 100, and plot the accuracy against that self-reported score. If the model claims 95% confidence on answers that your team scores as wrong, kill the autonomous visual agent and re-insert a human verification gate before you burn customer capital.