Key Takeaways

  • Hermes paired ESM protein embeddings with ChemBERTa small molecule embeddings via cross-attention, but attention maps exposed sequence memorization rather than physical binding.
  • State-of-the-art co-folding models exhibit the same failure mode: when researchers computationally plug a target cavity, the models continue shoving ligands into empty space.
  • Instead of learning interaction mechanics, Hermes fixated on variable strings of 10 to 12 amino acids away from active sites, assigning arbitrary nicknames to targets.
  • Leash Bio broke this memorization pattern by engineering point mutations into binding pockets to generate true negative experimental controls across chemical series.
  • The validation workflow follows the Pocket-Targeted Binding Validation Protocol.

The Pocket-Targeted Binding Validation Protocol

Machine learning models are lazy opportunists. When Ian Quigley and the team at Leash Bio built Hermes, they chose a direct setup. “The basic architecture of Hermes is two different embeddings,” Quigley explained. "One of them is an evolutionary scale modeling ESM embedding that's a protein embedding, and then we had a different embedding called ChemBERTa which is basically like a next token predictor chemistry. And then we just put them together with cross-attention and that was it."

Validation metrics looked clean, but the physics were broken. When the team inspected the cross-attention maps, Hermes was not paying attention to the functional binding pockets. As Quigley noted: “It was looking at really short sequences of amino acids that were away from the functional part of the protein and also away from part of the protein that looks similar across many proteins.”

The architecture had found a cheat code. “And we realized that Hermes was probably coming up with a nickname for the protein,” Quigley said. “It's not thinking about, oh, this is the functional part. It's like, I don't need to learn that. I can remember, you know, 10 or 12 amino acids that are different here than everyone else, and I can just memorize it.”

This behavior is not unique to Hermes. Leading co-folding tools fail the exact same sanity check. Quigley pointed out: “If you use current state of the art models to go see what the model will do, it will shove the small molecule in where the pocket used to be. Because it just remembers where it is.”

To strip away these sequence shortcuts and force the network to learn actual binding mechanics, Leash Bio built an experimental protocol that alters the target binding pocket directly:

  • Step 1: Baseline Wild-Type Screen: Screen the protein with an intact pocket against the DNA-encoded library to establish baseline enrichment and identify candidate binder populations.
  • Step 2: Competitive Displacement Screen: Introduce a known, characterized chemical competitor at high concentration into the assay and rerun the screen to identify all chemical series displaced from that specific pocket.
  • Step 3: Pocket Residue Identification and Mutagenesis: Inspect the structural amino acids within the binding pocket and mutate the DNA construct to introduce a bulky residue or alter a critical side-chain, physically breaking the pocket.
  • Step 4: Mutant Screen Verification: Rerun the screen against the mutant construct. Confirm that the chemical series booted out by the competitor also drop out in the mutant, while binding across non-pocket regions remains intact.

Retraining Hermes on these mutation datasets forced the model to abandon sequence shortcuts. As Quigley confirmed: "And so we went back and did train models with these additional mutations... and now Hermes will look at all of these things much more evenly."

When This Works (and When It Doesn't)

Applies when verifying whether chemical hits are authentically binding into a specific protein functional pocket rather than sticking non-specifically or binding to DNA barcodes. It gives machine learning models the negative counterfactuals they need to learn spatial chemistry rather than protein sequence identifiers.

However, the method breaks down when point mutations destabilize the overall tertiary fold. If introducing a bulky residue causes the entire protein to aggregate, precipitate, or misfold globally, binders will drop out due to structural collapse rather than pocket-specific disruption. It also fails on intrinsically disordered proteins or proteins with highly dynamic, cryptic pockets that lack clear structural coordinates.

What to Do With This

If you are building predictive models on biological or structured pairing data, run an attention inspection audit on your top 20 inference targets this week.

First, pull the attention weight maps across your sequence representations for your highest-scoring compound-protein pairs. If the weights light up on hypervariable terminal loops rather than documented active sites or conserved catalytic residues, stop celebrating your holdout test loss. Your model has built a lookup table of nicknames.

Next, build an automated validation script based on the protocol steps above. Simulate physical pocket disruption computationally or run targeted wet-lab mutagenesis screens on your primary target class. Rerun inference on the altered construct. If your model still predicts strong affinity despite the steric clash, penalize those weights immediately and add pocket-disrupted negative pairs into your training batches.