AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases
By Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Xiaodong Song, Bo Li. Neural Information Processing Systems, 2024
386 citations recorded by the source corpus.
LLM agents have demonstrated remarkable performance across various applications, primarily due to their advanced capabilities in reasoning, utilizing external knowledge and tools, calling APIs, and executing actions to interact with environments. Current agents typically utilize a memory module or a retrieval-augmented generation (RAG) mechanism, retrieving past knowledge and instances with similar embeddings from knowledge bases to inform task planning and execution. However, the reliance on unverified knowledge bases raises significant concerns about their safety and trustworthiness. To uncover such vulnerabilities, we propose a novel red teaming approach AgentPoison, the first backdoor attack targeting generic and RAG-based LLM agents by poisoning their long-term memory or RAG knowledge base. In particular, we form the trigger generation process as a constrained optimization to optimize backdoor triggers by mapping the triggered instances to a unique embedding space, so as to ensure that whenever a user instruction contains the optimized backdoor trigger, the malicious demonstrations are retrieved from the poisoned memory or knowledge base with high probability. In the meantime, benign instructions without the trigger will still maintain normal performance. Unlike conventional backdoor attacks, AgentPoison requires no additional model training or fine-tuning, and the optimized backdoor trigger exhibits superior transferability, in-context coherence, and stealthiness. Extensive experiments demonstrate AgentPoison's effectiveness in attacking three types of real-world LLM agents: RAG-based autonomous driving agent, knowledge-intensive QA agent, and healthcare EHRAgent. On each agent, AgentPoison achieves an average attack success rate higher than 80% with minimal impact on benign performance (less than 1%) with a poison rate less than 0.1%.
View Paper: https://www.semanticscholar.org/paper/b6948a9e8b3eec5a56a80c69727154fcd7ececce
Citations in the corpus
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (2024, unknown): “In line with the indirect prompt injection threat model (as opposed to our misuse setting), Chen et al. (2024) explored AgentPoison, a method for red-teaming LLM agents via memory or knowledge base poisoning, and Pelrine et al. (2023) observed that tool calling could facilitate jailbreaks for GPT…”
- Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (2024, unknown): “Memory poisoning involves injecting malicious or misleading data into a database (a memory unit or a RAG knowledge base) so that when this data is retrieved and processed later, it causes the agents to perform malicious actions (Xiang et al., 2024a; Chen et al., 2024).”
- Do Agents Dream of False Memories? Black-box Visual Attacks on Long-term Memory in Multimodal AI Agents (2026, unknown): “…operate exclusively in the text modality, constructing adversarial content through poisoned queries, manipulated reasoning chains, or trigger-response pairs, and either require direct write access to the memory store [16] or assume the attacker can craft text queries at interaction time [21].”
- ER-MIA: Black-Box Adversarial Memory Injection Attacks on Long-Term Memory-Augmented Large Language Models (2026, unknown): “…in memory-augmented or retrieval-augmented LLM systems (Dong et al., 2025; Chen et al., 2024; Zou et al., 2025), they typically assume agen-tic memory construction and task execution (Dong et al., 2025), trigger-based control (Chen et al., 2024), or access to a static corpora (Zou et al., 2025).”