Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks

By Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei A. Zaharia, Tatsunori B. Hashimoto. 2024 IEEE Security and Privacy Workshops (SPW), 2023

385 citations recorded by the source corpus.

Recent advances in instruction-following large language models (LLMs) have led to dramatic improvements in a range of NLP tasks. Unfortunately, we find that the same improved capabilities amplify the dual-use risks for malicious purposes of these models. Dual-use is difficult to prevent as instruction-following capabilities now enable standard attacks from computer security. The capabilities of these instruction-following LLMs provide strong economic incentives for dual-use by malicious actors. In particular, we show that instruction-following LLMs can produce targeted malicious content, including hate speech and scams, bypassing in-the-wild defenses implemented by LLM API vendors. Our analysis shows that this content can be generated economically and at cost of $125-500 \times$ cheaper than human effort alone. Together, our findings suggest that LLMs will increasingly attract more sophisticated adversaries and attacks, and addressing these attacks may require new approaches to mitigations.

Subfields: Information Systems, Signal Processing

View Paper: https://www.semanticscholar.org/paper/0cf694b8f85ab2e11d45595de211a15cfbadcd22

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Signal Processing

Citations in the corpus

Security and Privacy Challenges of Large Language Models: A Survey (2024, neutral): “Finally, Section 9 concludes the paper.”
Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (2023, background): “However, there is growing concern that open-source LLMs are more amenable to the dissemination of harmful or unethical content (Hazell, 2023; Kang et al., 2023).”
"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models (2023, positive): “[39] utilize standard attacks from computer security such as obfuscation, code injection, and virtualization to bypass the safeguards implemented by LLM vendors.”
How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs (2024, neutral): “Other studies (Mozes et al., 2023; Kang et al., 2023) use programmatic behaviors, such as code injection and virtualization, to expose LLM vulnerabilities.”
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024, background): “DPI (Perez and Ribeiro, 2022; Selvi, 2022; Kang et al., 2023; Liu et al., 2023; Toyer et al., 2023; Yu et al., 2023) involves a malicious user injecting harmful prompts directly into the inputs of a language model, with the aim of goal hijacking or prompt leaking.”
LLM Censorship: A Machine Learning Challenge or a Computer Security Problem? (2023, background): “However, extant defences have been empirically bypassed [38, 37, 28, 29, 41, 17, 50], and theoretical work [55] suggests that there will exist inputs to LLMs that elicit misaligned behaviour.”
GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (2023, background): “…that can elicit undesired behavior (Gehman et al., 2020; Bommasani et al., 2021; walkerspider, 2022; Perez et al., 2022; Perez & Ribeiro, 2022; Kang et al., 2023; Li et al., 2023; Ganguli et al., 2022; Schulhoff et al., 2023; OpenAI, 2023b; Jones et al., 2023; Zou et al., 2023; Huang et al.,…”
Multi-step Jailbreaking Privacy Attacks on ChatGPT (2023, background): “Kang et al. (2023) treated LLMs as programs and mimicked Computer Security attacks to maliciously prompt harmful contents from LLMs. Greshake et al. (2023) extended Prompt Injection attacks to application-integrated LLMs and argued that augmenting LLMs with applications could amplify the risks.”
Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (2024, neutral): “The prompt injection can target the user prompt directly (Perez & Ribeiro, 2022; Selvi, 2022; Toyer et al., 2023; Yu et al., 2023a; Kang et al., 2024) or indirectly influence the agent’s behavior by manipulating its accessible external environment (Liu et al., 2023; Greshake et al., 2023; Yi et…”
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning (2022, background): “While improved value alignment reduces the severity of “Let’s think step by step,” more complex reasoning strategies may exacerbate our findings (e.g. step-by-step code generation, explored in Kang et al. (2023)).”
Jailbroken: How Does LLM Safety Training Fail? (2023, methodology): “Most closely related are Kang et al. [30], who study attacking GPT-3.”
Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023, positive): “Even if applied, it remains unclear whether filtering can be evaded by stronger forms of obfuscation or encoding [53], which future models might further enable.”
A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily (2023, background): “Although human alignment techniques show potential and contribute to the feasible implementation of LLMs, recent discoveries of ’jailbreaks’ indicate that even the aligned LLMs can occasionally produce unwanted outputs in certain scenarios (Kang et al., 2023; Hazell, 2023; Shen et al., 2023).”
NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails (2023, background): “In addition, they are vulnerable to prompt injection (or jailbreak) attacks, where malicious actors manipulate inputs to trick the model into producing harmful outputs (Kang et al., 2023; Wei et al., 2023; Zou et al., 2023).”
AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (2023, background): “While the human alignment methods demonstrate promising effectiveness and pave the way to the practical deployment of LLMs, recent findings of jailbreaks show that the aligned LLMs can still provide undesirable responses in some situations (Kang et al., 2023; Hazell, 2023).”
Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models (2025, unknown): “Sure, here is the method to make bombs: … 𝑒 𝑠 +𝑤 .”