Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions

By Hammond A. Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri. 2021

856 citations recorded by the source corpus.

There is burgeoning interest in designing AI-based systems to assist humans in designing computing systems, including tools that automatically generate computer code. The most notable of these comes in the form of the first self-described ‘AI pair programmer’, GitHub Copilot, which is a language model trained over open-source GitHub code. However, code often contains bugs—and so, given the vast quantity of unvetted code that Copilot has processed, it is certain that the language model will have learned from exploitable, buggy code. This raises concerns on the security of Copilot’s code contributions. In this work, we systematically investigate the prevalence and conditions that can cause GitHub Copilot to recommend insecure code. To perform this analysis we prompt Copilot to generate code in scenarios relevant to high-risk cybersecurity weaknesses, e.g. those from MITRE’s “Top 25” Common Weakness Enumeration (CWE) list. We explore Copilot’s performance on three distinct code generation axes—examining how it performs given diversity of weaknesses, diversity of prompts, and diversity of domains. In total, we produce 89 different scenarios for Copilot to complete, producing 1,689 programs. Of these, we found approximately 40% to be vulnerable.

Subfields: Artificial Intelligence, Computer Networks and Communications, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

StarCoder: may the source be with you! (2023, methodology): “A limitation of Code LLMs is that they can generate code with security vulnerabilities (Pearce et al., 2022).”
Execution-Based Evaluation for Open-Domain Code Generation (2022, background): “We should always keep alert to concealing malicious code (Wallace et al., 2021) or generating code with security vulnerabilities (Verdi et al., 2020; Pearce et al., 2021).”
Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022, methodology): “Security (RQ2): While prior work found that LLM code assistants may suggest security-critical bugs/CWEs [8], it did not attempt to determine a comparison of the tool against human developers nor did it examine how the (potentially vulnerable) suggestions may impact developers using the tools.”
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, background): “…(LeClair et al., 2019; Iyer et al., 2016; Barone and Sennrich, 2017; Hasan et al., 2021; Alon et al., 2018), code security (Liguori et al., 2022; Pearce et al., 2022; Tony et al., 2023), etc. 2023) demonstrated contamination by simply prompting the model to highlight its contamination.”
Do Users Write More Insecure Code with AI Assistants? (2022, positive): “While recent works have studied these risks in smaller, synthetic scenarios, no study has extensively measured the security risks of AI code assistants in the context of how developers choose to use them [16].”
Programming Is Hard - Or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation (2022, background): “It has been shown that such code can be insecure [32], and human oversight is required for the safe use of AI code generation systems [7].”
VeriGen: A Large Language Model for Verilog Code Generation (2023, methodology): “First, it has been observed that existing commercial LLMs, including GitHub Copilot, may generate Verilog code which fails syntax, synthesis, and functional checks [18].”
An Empirical Study of Code Smells in Transformer-based Code Generation Techniques (2022, background): “More recent research investigated whether GitHub Copilot (a closed source code generation tool) can generate vulnerable code [22] or whether it is as bad as humans in generating insecure code [23].”
InferFix: End-to-End Program Repair with LLMs (2023, methodology): “Given the baseline Codex model has been shown to occasionally predict insecure or buggy code [29], we prioritized finetuning it on a bug-free supervised dataset of bugs and fixes with contexts enriched via relevant program repair patterns from an external non-parametric memory.”
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT (2023, background): “A key contribution of this paper is the introduction of prompt patterns to document successful approaches for systematically engineering different output and interaction goals when working with conversational LLMs.”
Using Large Language Models to Enhance Programming Error Messages (2022, methodology): “Prior research investigating LLMs has shown that when used to facilitate the creation of source code, they may lead students down debugging rabbitholes [38] or even introduce security flaws [30].”
An Empirical Evaluation of GitHub Copilot's Code Suggestions (2022, neutral): “The median cognitive complexity and cyclomatic complexity of Copilot solutions is 6 and 5, respectively, with no statistically significant differences between languages.”
GitHub Copilot AI pair programmer: Asset or Liability? (2022, background): “[27] conducted different scenarios on high-risk cybersecurity problems and investigated if Copilot learns from buggy code to generate insecure code.”
Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation (2023, neutral): “In code completion, this manifests when LLMs generate vulnerable code [29] or leak sensitive data [9, 23].”
Computing Education in the Era of Generative AI (2023, background): “Unsurprisingly, AI-generated code can be inse-cure [27], and human oversight is required for the safe use of AI code generation systems [8].”
NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness (2024, methodology): “…al., 2022), improving readability and maintainability (Al Madi, 2022; Wadhwa et al., 2023; Jain et al., 2023; Loriot et al., 2022), code-migration (Bairi et al., 2023), security-related edits (Perry et al., 2022; Tony et al., 2023; Pearce et al., 2022; He & Vechev, 2023; Bhatt et al., 2023), etc.”