OpenAI discovers 30 percent of popular AI coding test is broken

1 sourceFirst reported 12 days agoUpdated 12 days ago

AI summary: catch up in 20 seconds

OpenAI has discovered that about 30 percent of the tasks in the SWE-Bench Pro coding test are malfunctioning.
The SWE-Bench Pro test is commonly utilized to assess the programming abilities of AI models.
As a result of these findings, OpenAI is withdrawing its previous endorsement of the SWE-Bench Pro benchmark.
The issue raises concerns about the reliability of certain benchmarks used in evaluating AI coding skills.
OpenAI's review highlights the need for updates or replacements for current assessment tools in AI development.

Generated by AI from the 1 sources below · always verify with the originals

AI summary: catch up in 20 seconds

Summarising the coverage…

Generated by AI from the 1 sources below · always verify with the originals

All coverage (1)

OpenAI finds roughly 30 percent of popular AI coding test is brokenThe Decoder · 12 days ago

No comments yet. Be the first to join the discussion.

AI summary: catch up in 20 seconds

OpenAI has discovered that about 30 percent of the tasks in the SWE-Bench Pro coding test are malfunctioning.
The SWE-Bench Pro test is commonly utilized to assess the programming abilities of AI models.
As a result of these findings, OpenAI is withdrawing its previous endorsement of the SWE-Bench Pro benchmark.
The issue raises concerns about the reliability of certain benchmarks used in evaluating AI coding skills.
OpenAI's review highlights the need for updates or replacements for current assessment tools in AI development.

Generated by AI from the 1 sources below · always verify with the originals