InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents

By Qiusi Zhan, Zhixiang Liang, Zifan Ying, Daniel Kang. Annual Meeting of the Association for Computational Linguistics, 2024

443 citations recorded by the source corpus.

Recent work has embodied LLMs as agents, allowing them to access tools, perform actions, and interact with external content (e.g., emails or websites). However, external content introduces the risk of indirect prompt injection (IPI) attacks, where malicious instructions are embedded within the content processed by LLMs, aiming to manipulate these agents into executing detrimental actions against users. Given the potentially severe consequences of such attacks, establishing benchmarks to assess and mitigate these risks is imperative. In this work, we introduce InjecAgent, a benchmark designed to assess the vulnerability of tool-integrated LLM agents to IPI attacks. InjecAgent comprises 1,054 test cases covering 17 different user tools and 62 attacker tools. We categorize attack intentions into two primary types: direct harm to users and exfiltration of private data. We evaluate 30 different LLM agents and show that agents are vulnerable to IPI attacks, with ReAct-prompted GPT-4 vulnerable to attacks 24% of the time. Further investigation into an enhanced setting, where the attacker instructions are reinforced with a hacking prompt, shows additional increases in success rates, nearly doubling the attack success rate on the ReAct-prompted GPT-4. Our findings raise questions about the widespread deployment of LLM Agents. Our benchmark is available at https://github.com/uiuc-kang-lab/InjecAgent.

Subfields: Artificial Intelligence, Management Information Systems

View Paper: https://www.semanticscholar.org/paper/c8eee9766f0968e8f1b1be0731bc70b85be0ac97

Citations in the corpus

• Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (2024, unknown): “Although several bench-1 marks have been proposed to evaluate the security of LLM agents, such as InjecAgent (Zhan et al., 2024) and AgentDojo (Debenedetti et al., 2024), they are often limited by their scope, assessing either a single type of attack, i.e., Indirect Prompt Injection, or operating…”
• AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (2024, unknown): “A series of prior works have focused on the related setting of indirect attacks where a benign primary user’s agent is attacked via a malicious third-party provided resource (Ruan et al., 2023; Zhan et al., 2024; Debenedetti et al., 2024).”