Prompt Injection attack against LLM-integrated Applications

By Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Yanhong Zheng, Yang Liu. arXiv.org, 2023

827 citations recorded by the source corpus.

Large Language Models (LLMs), renowned for their superior proficiency in language comprehension and generation, stimulate a vibrant ecosystem of applications around them. However, their extensive assimilation into various services introduces significant security risks. This study deconstructs the complexities and implications of prompt injection attacks on actual LLM-integrated applications. Initially, we conduct an exploratory analysis on ten commercial applications, highlighting the constraints of current attack strategies in practice. Prompted by these limitations, we subsequently formulate HouYi, a novel black-box prompt injection attack technique, which draws inspiration from traditional web injection attacks. HouYi is compartmentalized into three crucial elements: a seamlessly-incorporated pre-constructed prompt, an injection prompt inducing context partition, and a malicious payload designed to fulfill the attack objectives. Leveraging HouYi, we unveil previously unknown and severe attack outcomes, such as unrestricted arbitrary LLM usage and uncomplicated application prompt theft. We deploy HouYi on 36 actual LLM-integrated applications and discern 31 applications susceptible to prompt injection. 10 vendors have validated our discoveries, including Notion, which has the potential to impact millions of users. Our investigation illuminates both the possible risks of prompt injection attacks and the possible tactics for mitigation.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/db4cf9f6a653d5c15973e836c800ea47743251ae

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyuan Jia
2. Neil Zhenqiang Gong
3. Jiahao Yu
4. Xinwei Wu
5. Tianhao Shen
6. Weilong Dong
7. Chuang Liu
8. Qiusi Zhan

In Information Systems

1. Jinyuan Jia
2. Neil Zhenqiang Gong
3. Binghui Wang
4. Le Zhang
5. Hao Fu

Citations in the corpus

• Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models (2023, neutral): “While the same positive correlation is also observed in code attacks, the corresponding Pearson correlation coefficient is lower than that of text attacks.”
• Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (2024, neutral): “…injection can target the user prompt directly (Perez & Ribeiro, 2022; Selvi, 2022; Toyer et al., 2023; Yu et al., 2023a; Kang et al., 2024) or indirectly influence the agent’s behavior by manipulating its accessible external environment (Liu et al., 2023; Greshake et al., 2023; Yi et al., 2023).”
• The rise and potential of large language model based agents: a survey (2025, neutral): “There are also some relevant attack methods such as dataset poisoning [520], backdoor attacks [521,522], and prompt-specific attacks [523,524], with the potential to induce LLMs to generate toxic content [525–527].”
• Formalizing and Benchmarking Prompt Injection Attacks and Defenses (2023, background): “Some prompt injection attacks [27,36] consider another scenario where the victim is the LLM-integrated Application, and a malicious user of the LLM-integrated Application is an attacker and leaks private information of the LLM-integrated Application.”
• Large Language Model Alignment: A Survey (2023, background): “Backdoor attacks on aligned models often utilize Prompt Injection techniques (Liu et al., 2023e; Zhao et al., 2023a; Greshake et al., 2023; Kandpal et al., 2023), where the prompt itself serves as the trigger, eliminating the need for external inputs.”
• PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts (2023, neutral): “In the era of large language models, Wang et al. (2023b) evaluated ChatGPT and other LLMs with respect to their robustness to adversarial samples and out-of-distribution (OOD) samples.”
• InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024, methodology): “Strategies include fine-tuning the LLM with attack cases (Yi et al., 2023; Piet et al., 2023), replacing command words with encoded versions and instructing the LLM to accept only these encoded commands (Suo, 2024), and separating prompts and data into two channels using a secure front-end for…”
• GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts (2023, background): “Furthermore, their widespread use has made them targets for adversarial attacks, including backdoor attacks [5, 32, 41], prompt injection [23,36,49], and data poisoning [34,43,73].”
• The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024, background): “Defenses for Prompt Injection For prompt injection on closed-domain tasks (Section 3.2), recent work has advocated for teaching a model to treat third-party user inputs as data, not as instructions (Chen et al., 2024; Willison, 2023; Zverev et al., 2024; Yi et al., 2023; Liu et al., 2023).”
• Knowledge Conflicts for LLMs: A Survey (2024, neutral): “Prompt injection attack (Liu et al., 2023b; Greshake et al., 2023; Yi et al., 2023; Xu et al., 2024a) is one such technique, where models may inadvertently spread misinformation if they use deceptive inputs (Pan et al., 2023b; Xu et al., 2023).”
• Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models (2023, neutral): “These adversarial textual inputs and prompt injections (Liu et al., 2023b; Perez & Ribeiro, 2022) exhibit high transferability, enabling them to bypass the safety guards of different LLMs (Wei et al., 2023).”