Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models
By Jingwei Yi, Yueqi Xie, Bin Zhu, Keegan Hines, Emre Kıcıman, Guangzhong Sun, Xing Xie, Fangzhao Wu. Knowledge Discovery and Data Mining, 2023
305 citations recorded by the source corpus.
The integration of large language models (LLMs) with external content has enabled applications such as Microsoft Copilot but also introduced vulnerabilities to indirect prompt injection attacks. In these attacks, malicious instructions embedded within external content can manipulate LLM outputs, causing deviations from user expectations. To address this critical yet under-explored issue, we introduce the first benchmark for bindirect prompt injection attacks, named BIPIA, to assess the risk of such vulnerabilities. Using BIPIA, we evaluate existing LLMs and find them universally vulnerable. Our analysis identifies two key factors contributing to their success: LLMs' inability to distinguish between informational context and actionable instructions, and their lack of awareness in avoiding the execution of instructions within external content. Based on these findings, we propose two novel defense mechanisms -- boundary awareness and explicit reminder -- to address these vulnerabilities in both black-box and white-box settings. Extensive experiments demonstrate that our black-box defense provides substantial mitigation, while our white-box defense reduces the attack success rate to near-zero levels, all while preserving the output quality of LLMs. We hope this work inspires further research into securing LLM applications and fostering their safe and reliable use. Our code is available at https://github.com/microsoft/BIPIA.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/6d465be006615460d41060f9f5068d51fc1f46b1
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jinyuan Jia
- Neil Zhenqiang Gong
- Qiusi Zhan
- Daniel Kang
- Zifan Ying
- Zhixiang Liang
- Binghui Wang
- Yang Zhang
Citations in the corpus
- StruQ: Defending Against Prompt Injection with Structured Queries (2024, unknown): “Our findings align with those in Yi et al. [39], Piet et al. [10]: Vulnerability to prompt injection stems from models’ ability to follow instructions and inability to distinguish between instructions and data.”
- Formalizing and Benchmarking Prompt Injection Attacks and Defenses (2023, background): “Other defenses against prompt injection attacks: We note that several recent studies [17,38,55] proposed other defenses against prompt injection attacks.”
- InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024, methodology): “On the other hand, IPI (Abdelnabi et al., 2023; Yi et al., 2023) entails an attacker injecting harmful prompts into exter-nal content that an LLM is expected to retrieve, with the goal of diverting benign user instructions.”
- ER-MIA: Black-Box Adversarial Memory Injection Attacks on Long-Term Memory-Augmented Large Language Models (2026, unknown): “Prior work has shown that prompt injection can be performed indirectly, through content retrieved from external sources rather than explicit user prompts (Greshake et al., 2023; Yi et al., 2025).”