Measuring Coding Challenge Competence With APPS
By Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song and 1 more. 2021
1,191 citations recorded by the source corpus.
While programming is one of the most broadly applicable skills in modern society, modern machine learning models still cannot code solutions to basic problems. Despite its importance, there has been surprisingly little work on evaluating code generation, and it can be difficult to accurately assess code generation performance rigorously. To meet this challenge, we introduce APPS, a benchmark for code generation. Unlike prior work in more restricted settings, our benchmark measures the ability of models to take an arbitrary natural language specification and generate satisfactory Python code. Similar to how companies assess candidate software developers, we then evaluate models by checking their generated code on test cases. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. We fine-tune large language models on both GitHub and our training set, and we find that the prevalence of syntax errors is decreasing exponentially as models improve. Recent models such as GPT-Neo can pass approximately 20% of the test cases of introductory problems, so we find that machine learning models are now beginning to learn how to code. As the social significance of automatic code generation increases over the coming years, our benchmark can provide an important measure for tracking advancements.
Subfields: Artificial Intelligence, Information Systems, Software, Evaluation & Benchmarks, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Jianfeng Gao
- Luke S. Zettlemoyer
- Yoshua Bengio
- Quoc V. Le
- Graham Neubig
- Sergey Levine
In Information Systems
- Percy Liang
- Graham Neubig
- Charles Sutton
- Xinyun Chen
- Chun Xia
- Lingming Zhang
- Koushik Sen
- Luke S. Zettlemoyer
In Software
- Josh Tenenbaum
- Chun Xia
- Lingming Zhang
- Koushik Sen
- Pushmeet Kohli
- Percy Liang
- Armando Solar-Lezama
- Michihiro Yasunaga
In Evaluation & Benchmarks
- Ashish Sabharwal
- Chitta Baral
- Luke S. Zettlemoyer
- Peter Clark
- Mohit Bansal
- Percy Liang
- Pieter Abbeel
- Dan Roth
In Code Generation & Program Synthesis
- Charles Sutton
- Graham Neubig
- Augustus Odena
- Baptiste Rozière
- Guillaume Lample
- Josh Tenenbaum
- Chun Xia
- Lingming Zhang
Citations in the corpus
- Competition-level code generation with AlphaCode (2022, methodology): “Early work using program synthesis for competitive programming has shown that large transformer models can achieve low single-digit solve rates (Chen et al., 2021; Hendrycks et al., 2021), but could not yet reliably generate solutions for the vast majority of problems.”
- Program Synthesis with Large Language Models (2021, methodology): “Such datasets include the Natural Program Synthesis (NAPS) dataset [Zavershynskyi et al., 2018], the Search-based Pseudocode to Code (SPoC) dataset [Kulal et al., 2019], the APPS dataset [Hendrycks et al., 2021], the PROGRES dataset [Alet et al., 2021], and the CodeNet dataset [Puri et al., 2021].”
- Evaluating Large Language Models Trained on Code (2021, methodology): “Most relevant to our evaluation work is the APPS (Hendrycks et al., 2021) benchmark for measuring functional correctness based on problems from the competitive programming website Codeforces.”
- Code Llama: Open Foundation Models for Code (2023, methodology): “With the same training process, our larger models outperform their smaller counterparts on almost every metric from HumanEval, MBPP and APPS (Table 2, 3).”
- Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, methodology): “, 2020), synthesize code from docstrings (Hendrycks et al., 2021; Chen et al., 2021), solve math problems (Cobbe et al.”
- MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, methodology): “There are a number of existing single-language NL2Code benchmarks [8], [9], [10].”
- CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023, methodology): “However, similar to MBPP [2] and APPS [13], it only consists of handcrafted programming problems in Python, thus cannot be directly applied to systematically evaluate the performance of multilingual code generation.”
- SantaCoder: don't reach for the stars! (2023, positive): “This dataset was also decontaminated by removing files that contained test-samples from the following benchmarks: HumanEval (Chen et al., 2021), APPS (Hendrycks et al., 2021), MBPP (Austin et al., 2021) and MultiPL-E (Cassano et al., 2022).”
- Execution-Based Evaluation for Open-Domain Code Generation (2022, methodology): “The predominant approach is to create hand-written test cases (Chen et al., 2021; Austin et al., 2021; Hendrycks et al., 2021; Lai et al., 2022; Huang et al., 2022) for the canonical code solutions.”
- The Stack: 3 TB of permissively licensed source code (2022, background): “Evaluation of code LLMs Code LLMs are frequently evaluated on text-to-code benchmarks, in which models are tasked with completing code snippets from natural language descriptions (Austin et al., 2021; Hendrycks et al., 2021; Chen et al., 2021; Lai et al., 2022).”
- DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation (2022, background): “Consequently, most benchmarks with test cases focus on competition/interview/programming challenges (Hendrycks et al., 2021; Li et al., 2022), because these are the only applications where a lot of test cases are already available.”
- CodeT: Code Generation with Generated Tests (2022, background): “, 2021), APPS (Hendrycks et al., 2021), and CodeContests (Li et al.”
- Multi-lingual Evaluation of Code Generation Models (2022, background): “Many other evaluation datasets can be considered for the conversion to multi-lingual counterparts such as APPS (Hendrycks et al., 2021) and CodeContest (Li et al.”
- Natural Language to Code Translation with Execution (2022, background): “Multiple benchmarks have been proposed to evaluate code model performance (Miceli Barone and Sennrich, 2017; Yin et al., 2018; Hendrycks et al., 2021; Lu et al., 2021, inter alia ).”
- CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks (2021, methodology): “Our initial focus is on text to code generation and we plan to fine-tune GPT-Neo on a combination of the APPS dataset [44] and CodeNet, since CodeNet also contains problem specifications and test inputs.”
- LLM-Assisted Code Cleaning For Training Accurate Code Generators (2023, methodology): “APPS (Hendrycks et al., 2021).”
- Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks (2023, methodology): “We leverage these properties to design LLM attacks that are described below.”
- CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023, methodology): “Large language models (LLMs) [Chen et al., 2021, Wang et al., 2021b, Nijkamp et al., 2023b] have recently demonstrated remarkable success in a broad set of downstream tasks in the code domain [Husain et al., 2019, Lu et al., 2021, Hendrycks et al., 2021].”