Key Takeaways

  • In late 2023, Leash Bio attempted supervised fine-tuning on OpenAI and Anthropic models because frontier LLMs failed basic SMILES string manipulations.
  • SMILES encodes a molecular graph into a single string of text, meaning different traversal starting points yield different valid representations for the same chemical compound.
  • Around February 2024, frontier foundation models abruptly crossed a capability threshold, handling organic chemistry strings without specialized fine-tuning.
  • In Leash Bio's Minos benchmark, stripping SMILES strings out of agent context windows caused iterative molecular search performance to drop immediately.
  • Modern foundation models actively parse graph structures in context rather than relying on surface labels or random exploration patterns.

The Failure of Chemistry Fine-Tuning

In late 2023, the engineering team at Leash Bio hit a wall with standard foundation models. They wanted models to read and edit molecules using SMILES (Simplified Molecular Input Line Entry System), the standard text format chemists use to encode molecules. It did not work.

“We had started an internal project to do supervised fine tuning on some of these language models to get them to understand smiles,” says Ian Quigley of Leash Bio. “Because none of the existing frontier models were any good at it. So like none of the Anthropic or OpenAI models at the time could even convert smiles to non-canonical smiles very effectively.”

The problem comes down to geometry. A molecule is an interconnected graph of atoms and bonds. SMILES forces that graph into a linear text string by traversing bonds in order. “A smiles is basically a graph traversal across the molecule written down into a one-dimensional string,” Quigley explains. “And so you can imagine starting the graph traversal at a different point in the molecule. And so there's multiple smiles ways of representing the same molecule.”

To convert a canonical SMILES string to a non-canonical one, a model cannot just predict the next character based on surface text patterns. It has to reconstruct the ring structures, track atom connectivity, select a new starting node, and write a new valid path. In late 2023, every base model failed this task.

The February 2024 Shift and the Context Test

Then, frontier models changed.

“And nobody could do it,” Quigley says. “And then around February, like a lot of other things, it became clear that they could start to actually work with chemistry in a meaningful way, which is a reason we started going in this direction.”

Without announced chemistry fine-tuning from major labs, general frontier models crossed a capability threshold. They began interpreting the grammar of SMILES strings directly.

To test whether models were truly reasoning about chemistry rather than making lucky guesses, Leash Bio evaluated them inside Minos. Minos is a benchmark built on pre-measured wet lab assay data, letting AI agents test drug candidates against real physical results with zero latency. Quigley set up iterative search loops: an agent inspects molecules, reviews binding scores, and proposes modifications for the next round.

“Like these are the tasks that we're having these things do right now,” Quigley says. “Like they're iterative tasks where they're looking at molecules and they are learning something about those molecules to make choices to make better molecules next.”

Quigley then ran an ablation test: he stripped the SMILES strings out of the prompt, leaving only arbitrary identifiers and binding scores. The models collapsed.

“And we remove the information about the molecules, like these decisions really suffer,” Quigley notes. “So they're learning something. Or at least they're using it. We're not changing weights here, but they're definitely using the information just to make future choices.”

What to Do With This

Before spending capital fine-tuning a custom model on domain-specific syntax, run an ablation test on current frontier models. Feed your structured strings (whether ASTs, CAD formats, or chemical representations) into an iterative prompt loop, then strip the syntax while keeping the performance metrics identical. If the model with syntax beats the model with blinded IDs, you already have in-context domain reasoning. Stop training custom weights and build your evaluation environment instead.