You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion

By Roei Schuster, Congzheng Song, Eran Tromer, Vitaly Shmatikov. 2020

223 citations recorded by the source corpus.

Code autocompletion is an integral feature of modern code editors and IDEs. The latest generation of autocompleters uses neural language models, trained on public open-source code repositories, to suggest likely (not just statically feasible) completions given the current context. We demonstrate that neural code autocompleters are vulnerable to data- and model-poisoning attacks. By adding a few specially-crafted files to the autocompleter's training corpus, or else by directly fine-tuning the autocompleter on these files, the attacker can influence its suggestions for attacker-chosen contexts. For example, the attacker can "teach" the autocompleter to suggest the insecure ECB mode for AES encryption, SSLv3 for the SSL/TLS protocol version, or a low iteration count for password-based encryption. We moreover show that these attacks can be targeted: an autocompleter poisoned by a targeted attack is much more likely to suggest the insecure completion for certain files (e.g., those from a specific repo). We quantify the efficacy of targeted and untargeted data- and model-poisoning attacks against state-of-the-art autocompleters based on Pythia and GPT-2. We then discuss why existing defenses against poisoning attacks are largely ineffective, and suggest alternative mitigations.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Safety, Robustness & Interpretability, Code Generation & Program Synthesis, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Safety, Robustness & Interpretability

In Code Generation & Program Synthesis

In Signal Processing

Citations in the corpus

Program Synthesis with Large Language Models (2021, background): “[2021] provide a detailed overview of risks and potential harms of large language models over code, discussing potential concerns that include over-reliance on generated outputs, misalignment, poisoning attacks [Schuster et al., 2020], and others.”
Evaluating Large Language Models Trained on Code (2021, background): “Similarly, the public data should generally be treated as untrusted, as previous work (Goldblum et al., 2021; Schuster et al., 2020) has found that attackers may be able to corrupt training data to trigger speciﬁc model behaviors at runtime.”
Poisoning Language Models During Instruction Tuning (2023, background): “Wallace et al. (2020) 2020; Schuster et al., 2021; Yang et al., 2021) or directly manipulating the model weights (Kurita et al., 2020).”
Unsolved Problems in ML Safety (2021, background): “One way to create a backdoor is to directly inject the backdoor into a model’s weights [154, 89], but they can also be injected by adding poisoned data into the training or pretraining data [156].”
Backdoors in Neural Models of Source Code (2020, background): “Recent concurrent work [32] demonstrated the vulnerability of code completion models to backdoors, and the shortcomings of existing defenses.”
Poisoning Web-Scale Training Datasets is Practical (2023, background): “, a poisoning attack [6]) suffices to introduce targeted mistakes in model behavior [14, 15, 60, 76].”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “In this section, we examine the range of existing defense methods against various attacks and vulnerabilities associated with LLMs 1 .”
Poisoning the Unlabeled Dataset of Semi-Supervised Learning (2021, background): “Similar trends have recently been observed in other domains; for example, neural language models trained on unlabeled data scraped from the Internet can be poisoned to perform targeted mispredictions [49].”
Security and Privacy Challenges of Large Language Models: A Survey (2024, unknown): “This involves reviewing existing studies in the broad area of security, privacy, LLMs, and their intersections, and notably, highlighting future research directions to design novel evaluation protocols and attack defense mechanisms tailored to the evolving landscape of LLMs.”
AI-Driven Development Is Here: Should You Worry? (2022, background): “Subtle security holes can easily persist even in high-quality, high-volume repositories (consider the OpenSSH Heartbleed incident), and recent work showed how deep learning models can learn vulnerable code and inject it during autocompletion [Schuster et al., 2020].”
Contrastive Code Representation Learning (2020, background): “Yefet et al. (2020) and Schuster et al. (2021) ﬁnd accuracy degrades ∗ equal contribution Figure 1: Robust code clone detection: On source code, RoBERTa is not robust to simple label-preserving code edits like renaming variables.”
PoisonedEncoder: Poisoning the Unlabeled Pre-training Data in Contrastive Learning (2022, background): “Examples include recommender systems [19, 21, 27, 36, 60], federated analytics [7, 9, 10, 18, 20, 39, 58], search engines [33], as well as natural language models [47, 48].”