SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
By Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Kexin Pei, Ofir Press, Karthik Narasimhan. 2023
3,028 citations recorded by the source corpus.
Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities. We find real-world software engineering to be a rich, sustainable, and challenging testbed for evaluating the next generation of language models. To this end, we introduce SWE-bench, an evaluation framework consisting of $2,294$ software engineering problems drawn from real GitHub issues and corresponding pull requests across $12$ popular Python repositories. Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue. Resolving issues in SWE-bench frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously, calling for models to interact with execution environments, process extremely long contexts and perform complex reasoning that goes far beyond traditional code generation tasks. Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is able to solve a mere $1.96$% of the issues. Advances on SWE-bench represent steps towards LMs that are more practical, intelligent, and autonomous.
Subfields: Artificial Intelligence, Computer Networks and Communications, Information Systems, Evaluation & Benchmarks, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Noah A. Smith
- Luke S. Zettlemoyer
- Percy Liang
- Joseph E. Gonzalez
- Mohit Bansal
- Graham Neubig
- Yejin Choi
- Jianfeng Gao
In Computer Networks and Communications
- Joseph E. Gonzalez
- Ion Stoica
- Armando Solar-Lezama
- Shin Yoo
- Noah A. Smith
- Chun Xia
- Koushik Sen
- Mark Harman
In Information Systems
In Evaluation & Benchmarks
- Luke S. Zettlemoyer
- Chun Xia
- Lingming Zhang
- Noah A. Smith
- Armando Solar-Lezama
- Ranjay Krishna
- Mark Harman
- Naman Jain
In Code Generation & Program Synthesis
- Armando Solar-Lezama
- Noah A. Smith
- Chun Xia
- Koushik Sen
- Mark Harman
- Luke S. Zettlemoyer
- Yushi Hu
- Jiawei Liu
Citations in the corpus
- From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (2024, background): “Many have considered task-based evaluation such as IFEval [53], SWE-Bench [22], or AgentBench [28].”
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “…engineering tasks, such as JuICe (Agashe et al., 2019), APIBench (Patil et al., 2023), RepoBench (Liu et al., 2023c), ODEX (Wang et al., 2022b), SWE-Bench (Jimenez et al., 2023), GoogleCodeRepo (Shrivastava et al., 2023), RepoEval (Zhang et al., 2023a), and Cocomic-Data (Ding et al., 2022).”
- CodeMind: A Framework to Challenge Large Language Models for Code Reasoning (2024, methodology): “…(Wang et al., 2022; Athiwaratkun et al., 2022; Chen et al., 2021; Liu et al., 2023; Gu et al., 2024; Zheng et al., 2023; Cassano et al., 2022; Jimenez et al., 2023; Du et al., 2023; Odena et al., 2021; Puri et al., 2021; Ahmad et al., 2021), we chose the programs in HumanEval (Chen et al.,…”
- τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment (2025, neutral): “Following a long line of research into language agents [25, 27, 10, 13, 21], LLM tool use [24, 20, 9], and task-oriented dialog [5, 4, 1, 22, 6, 7, 8], τ -bench [26] is a recently introduced benchmark to measure the reliability of language agents in multi-turn task-oriented conversations such as…”
- Agentless: Demystifying LLM-based Software Engineering Agents (2024, methodology): “We evaluate A GENTLESS and baselines using the popular SWE-bench dataset [44] to test the ability to solve real-world software engineering issues.”
- Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (2023, background): “More recently, researchers have created a more realistic code synthesis benchmark by collecting GitHub issues along with the corresponding code base together with tests to measure the ability of LLMs to perform real-world software engineering tasks [28].”
- Large Language Models for Software Engineering: Survey and Open Problems (2023, background): “It also proposed 4 types of features for generative AI: AI documentation, model uncertainty, attention distribution, and social trans-parency (i.e., making visible the socio-organizational factors that govern the use of AI).”
- DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence (2024, neutral): “SWE-bench is a comprehensive benchmark designed to evaluate the performance of large language models in addressing real-world software issues sourced from GitHub.”
- MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (2024, neutral): “…context windows enable LLMs to unlock a multitude of complex real-world applications, such as repository-level code understanding [BSK + 24, JYW + 23, POC + 23], long-document question-answering [CPG + 23, LZD + 24], self-play reasoning [Ope24], extreme-label in-context learning [LZD +…”
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling (2024, neutral): “This ability to amplify weaker models extends to the challenging SWE-bench Lite dataset of real-life GitHub issues [32], where the current single-sample state-of-the-art (SOTA), achieved by a mixture of GPT-4o and Claude 3.5 Sonnet, is 43% [1].”
- CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges (2024, neutral): “Besides, some benchmarks (Li et al., 2022) contain algorithm questions.”
- GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models (2025, neutral): “5’s ability to complete real-world coding tasks, we evaluate it on two challenging benchmarks, SWE-bench Verified [20] and Terminal-Bench [35].”
- OpenHands: An Open Platform for AI Software Developers as Generalist Agents (2024, neutral): “They are moving towards performing complex tasks such as developing software (Jimenez et al., 2024), navigating real-world websites (Zhou et al., 2023a), doing household chores (Ahn et al., 2022), or even performing scientific research (Boiko et al., 2023; Tang et al., 2024a).”
- Kimi K2: Open Agentic Intelligence (2025, neutral): “For coding, we adopt LiveCodeBench v6 [32](questions from August 2024 to May 2025), OJBench [78], MultiPL-E [6], SWE-bench Verified [33, 85], TerminalBench [72], Multi-SWE-bench [87], SWE-Lancer [51], PaperBench [66], and Aider-Polyglot [17].”
- Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2024, neutral): “Multimodal Large Language Models (MLLMs) [1, 4, 15, 34, 42, 49, 49, 78], which integrate language and vision, exhibit strong capacities to think and reason in open-ended dialog and practical tasks like web agents [21, 28, 33, 35].”
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (2024, neutral): “SWE-bench (Jimenez et al., 2024) tasks models to solve real-world pull requests from open-source repositories.”
- A Survey on Large Language Models for Code Generation (2024, neutral): “CodeBLEU [52] pass@k [56] pass-ratio@n [53] ICE-Score [54] B. Benchmarks HumanEval [56] ClassEval [57] SWE-bench [58] BigCodeBench [59] VI. LLMs’ Applications”
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024, background): “Benchmarks like SWE-bench [29] are built to evaluate the performance of a code agent framework (e.g., iterated prompting, real-time environment interaction, and long-context exploration).”