Why we no longer evaluate SWE-bench Verified

In recent years, the software development community has been grappling with the challenges of accurately measuring the progress of frontier coding practices. One of the most prominent tools used for this purpose, SWE-bench Verified, has come under scrutiny due to its increasing contamination and inability to accurately reflect the advancements in the field. Our comprehensive analysis has revealed significant flaws in the testing framework and evidence of training leakage, which have led us to recommend a switch to SWE-bench Pro as a more reliable alternative.

SWE-bench Verified was initially designed to provide a standardized and objective measure of coding proficiency and efficiency. It aimed to assess developers' ability to write clean, efficient, and maintainable code by evaluating their performance on a set of predefined benchmarks. However, as the tool has gained popularity, it has become increasingly apparent that it is not living up to its original intentions.

One of the primary issues with SWE-bench Verified is the flawed nature of its tests. These tests often lack the sophistication and real-world relevance required to accurately gauge a developer's skills. In many cases, the problems presented in the benchmarks are contrived or overly simplistic, leading to results that do not accurately reflect a developer's ability to tackle complex, real-world coding challenges.

Moreover, our analysis has uncovered evidence of training leakage within the SWE-bench Verified framework. This refers to the practice of developers using information from the training materials to gain an unfair advantage in the benchmark tests. As a result, the scores achieved by these developers do not truly represent their coding abilities but rather their familiarity with the specific questions and problems included in the training materials.

The impact of these flaws on the credibility of SWE-bench Verified is significant. As the tool is widely used to evaluate coding progress and compare developers' skills, its inaccuracies can lead to misguided assessments and decisions. For instance, organizations relying on SWE-bench Verified to identify top talent may inadvertently overlook developers with genuine coding abilities who struggle with the contrived problems or lack access to the specific training materials.

In light of these concerns, we strongly recommend that the software development community consider switching to SWE-bench Pro as a more reliable and accurate alternative. SWE-bench Pro addresses the shortcomings of its Verified counterpart by offering a more robust and realistic set of benchmarks that better reflect the challenges developers face in real-world projects.

Furthermore, SWE-bench Pro has implemented stricter measures to prevent training leakage, ensuring that the results obtained are a true reflection of a developer's coding abilities. By adopting SWE-bench Pro, the community can gain greater confidence in the accuracy of coding evaluations and make more informed decisions about talent identification and development.

In conclusion, the increasing contamination and mismeasurement of frontier coding progress by SWE-bench Verified have raised serious concerns within the software development community. Our analysis has revealed significant flaws in the tool's testing framework and evidence of training leakage, which have undermined its credibility. As a result, we urge organizations and individuals to consider transitioning to SWE-bench Pro, a more reliable and accurate alternative that better serves the needs of the community. By doing so, we can ensure that coding evaluations remain a fair and objective measure of a developer's skills and contribute to the continued advancement of the field.