Do Users Write More Insecure Code with AI Assistants?

By Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh. Conference on Computer and Communications Security, 2022

397 citations recorded by the source corpus.

AI code assistants have emerged as powerful tools that can aid in the software development life-cycle and can improve developer productivity. Unfortunately, such assistants have also been found to produce insecure code in lab environments, raising significant concerns about their usage in practice. In this paper, we conduct a user study to examine how users interact with AI code assistants to solve a variety of security related tasks. Overall, we find that participants who had access to an AI assistant wrote significantly less secure code than those without access to an assistant. Participants with access to an AI assistant were also more likely to believe they wrote secure code, suggesting that such tools may lead users to be overconfident about security flaws in their code. To better inform the design of future AI-based code assistants, we release our user-study apparatus to researchers seeking to build on our work.

Subfields: Computer Science Applications, Information Systems, Safety Research

View Paper: https://www.semanticscholar.org/paper/ce3f027b68dad014a58aa35f52380932c8d0b209

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems

In Safety Research

Citations in the corpus

Parsel: A Unified Natural Language Framework for Algorithmic Reasoning (2022, background): “Yet, these models perform worse the more unusual the desired functions are, and recent research suggests that people using these language models are more likely to introduce buggy code (Perry et al., 2022), although this is not yet conclusive (Sandoval et al., 2022).”
The Metacognitive Demands and Opportunities of Generative AI (2023, background): “…meta-d ’ estimates How do aspects of GenAI output (e.g., the speed at which it is produced, its verbal fluency in the case of text) serve as heuristic cues that influence users’ confidence in the output and their evaluation ability, as well as the amount of effort they invest into evaluation?”
Computing Education in the Era of Generative AI (2023, background): “Perry et al. recently examined whether novices using AI code generation tools wrote more secure code, finding that novices consistently wrote insecure code with specific high vulnerabilities in string encryption and SQL injection [29].”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “Meanwhile, Perry et al. [2022] find that humans miss security vulnerabilities introduced by LLM code assistants.”
NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness (2024, background): “…al., 2022), improving readability and maintainability (Al Madi, 2022; Wadhwa et al., 2023; Jain et al., 2023; Loriot et al., 2022), code-migration (Bairi et al., 2023), security-related edits (Perry et al., 2022; Tony et al., 2023; Pearce et al., 2022; He & Vechev, 2023; Bhatt et al., 2023), etc.”
Repository-Level Prompt Generation for Large Language Models of Code (2022, background): “Blind trust in these systems may lead to potential negative impact, as there might be cases where the generated code is insecure (Perry et al., 2022) or contains sensitive information.”