Key Takeaways

  • A single researcher working in Ian Quigley's basement generated a four-billion-measurement dataset using DNA-encoded chemical libraries.
  • Across 2,000 competing teams in the BELKA Kaggle competition, not a single model beat random chance on an out-of-distribution chemical library.
  • Machine learning models scored well only when test molecules shared building blocks with training data, relying on memorization over physical rules.
  • Simple SMILES text strings outperformed fancy graph architectures, proving that model complexity cannot compensate for data distribution gaps.

The Basement Dataset That Stumped 2,000 Teams

Most founders assume drug discovery needs an army of wet-lab technicians and millions of dollars in robotic infrastructure. Leash Bio proved otherwise. Ian Quigley set up a workspace in his own home to test chemical scale. “And so we released a data set that was about four billion physical interactions and it was all measured by a single person in the basement of my house,” Quigley recalled.

Leash took that four-billion-row dataset and launched the BELKA competition on Kaggle. The premise seemed simple: train a model on small molecules, then predict whether those molecules bind to specific disease target proteins. Over 2,000 machine learning teams joined the race, throwing every conceivable architecture at the benchmark.

Then came the evaluation.

The Three-Tier Eval That Exposed Model Memorization

Leash organized the competition around three tiers of evaluation difficulty. The first tier tested molecules that shared building blocks with the training data. The second tier tested molecules that had different building blocks but retained the central chemical scaffold, a triazine hub. The third tier introduced a completely unseen chemical library built on different chemistry.

On tier one, the leaderboards lit up. Models accurately predicted binding events when they recognized the sub-units. On tier two, performance held up for certain targets where the shared triazine hub provided structural hints. But tier three broke everything.

“And then when you go to the most challenging eval, no one did better than chance,” Quigley said. “It seemed pretty clear that they might be able to get an idea through probably through memorization that if you have a very similar building block, it might behave similarly. And then also that you have something that's pretty far away, at that regime they simply could not do it.”

Two thousand machine learning teams did not build physics engines. They built lookup tables. If an algorithm only predicts outcomes on molecules built from the exact blocks it studied on Monday, it cannot invent new medicine on Friday.

Why Basic Strings Beat Complex Architectures

Participants tried custom graph neural networks, geometric deep learning, and bespoke chemical embeddings. None of them solved the generalization gap. In fact, Leash discovered that raw chemical text strings beat the complex setups.

“And we were grateful to find that the thing that we were doing, which are just basic SMILES, kind of beat everything else,” Quigley said. “It wasn't like a resounding thumping, but it was better enough for us to continue using that approach.”

Rather than despairing over the failure of tier three, Quigley found the result clarifying. “Some of it was kind of delight, which is like, wow, this is much harder than we thought. That's kind of great. That's kind of great because if this problem is like all the other ones that we're aware of, if you can get pretty good signal out of a scalable measurement regime, then you should just try to go get that.”

When a model fails to generalize across chemical families, spending three months tweaking attention heads is a waste of capital. The bottleneck is physical signal, not layer depth.

What to Do With This

Split your benchmark into three strict buckets tomorrow: one with identical components, one with shared skeletons, and one completely out-of-distribution holdout. If your model scores tank to random chance on the third bucket, stop modifying model weights. Strip your input representation down to the simplest possible baseline, like plain text, and invest your engineering time into generating new physical measurements instead.