Are aligned neural networks adversarially aligned?

By Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramèr and 1 more. 2023

387 citations recorded by the source corpus.

Large language models are now tuned to align with the goals of their creators, namely to be"helpful and harmless."These models should respond helpfully to user questions, but refuse to answer requests that could cause harm. However, adversarial users can construct inputs which circumvent attempts at alignment. In this work, we study adversarial alignment, and ask to what extent these models remain aligned when interacting with an adversarial user who constructs worst-case inputs (adversarial examples). These inputs are designed to cause the model to emit harmful content that would otherwise be prohibited. We show that existing NLP-based optimization attacks are insufficiently powerful to reliably attack aligned text models: even when current NLP-based attacks fail, we can find adversarial inputs with brute force. As a result, the failure of current attacks should not be seen as proof that aligned text models remain aligned under adversarial inputs. However the recent trend in large-scale ML models is multimodal models that allow users to provide images that influence the text that is generated. We show these models can be easily attacked, i.e., induced to perform arbitrary un-aligned behavior through adversarial perturbation of the input image. We conjecture that improved NLP attacks may demonstrate this same level of adversarial control over text-only models.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Improved Baselines with Visual Instruction Tuning (2023, background): “The LLaVA architecture is also lever-aged in different downstream tasks and domains, including region-level [8, 55] and pixel-level [25, 49] understanding, biomedical assistants [30], image generation [5], adversarial studies [6, 58].”
How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs (2024, neutral): “Previous safety studies, like those outlined in Carlini et al. (2023) and explored in Yu et al. (2023), have touched on such social engineering risks in LLMs.”
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (2024, background): “However, a growing body of work has shown that even the most performant LLMs are not adversarially aligned: it is often possible to elicit undesirable content by using so-called jailbreaking attacks (Mowshowitz, 2022; Carlini et al., 2024).”
Low-Resource Languages Jailbreak GPT-4 (2023, background): “Prior work has explored non-English jailbreaking attacks using adversarial prompts in a code-switching fashion [24], injecting the morally unhinged TranslatorBot prompt [9], and obfuscating the inputs with base64 encoding [44], Morse code, or argot languages like Pig Latin, leetspeak, and cipher…”
Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models (2023, background): “While the process of alignment reduces the potential misuse of powerful AI tools, preliminary investigations have begun to underscore the potential vulnerabilities and gaps present in existing safety frameworks (Carlini et al., 2023; Haller et al., 2023).”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “…the lenses of theoretical properties [Wolf et al., 2023, El-Mhamdi et al., 2022], decision theory [Casper, 2020], adversarial attacks [Perez et al., 2022a,b, Casper et al., 2023b, Ziegler et al., 2022, Carlini et al., 2023b], or rigorous evaluations [ARC, 2022, OpenAI, 2023, Shevlane et al., 2023].”
LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked (2023, background): “However, an emerging body of work has revealed that even aligned models can be manipulated into producing harmful content by prompt engineering [28, 23, 16], or employing more advanced techniques such as adversarial suffix attacks [28, 34, 4].”
SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (2023, positive): “Yet, despite this progress, a burgeoning literature has identified numerous failure modes—commonly referred to as jailbreaks — that bypass the alignment mechanisms implemented on modern LLMs [9, 10].”
FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts (2023, background): “(3) FigStep leverages a benign inciting text-prompt to stimulate the VLMs’ reasoning capabilities and instruct the VLMs to generate detailed responses following the content of the image-prompt.”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, positive): “Carlini et al. [208] recently show that NLP-based attacks are weak due to discrete optimization involved, but these results should not be interpreted as showing that these foundation models are robust.”
Safety Alignment Should Be Made More Than Just a Few Tokens Deep (2024, positive): “…researchers demonstrate that aligned models can still be made to respond to harmful requests via adversarially optimized inputs (Qi et al., 2023a; Carlini et al., 2023; Zou et al., 2023a; Chao et al., 2023; Andriushchenko et al., 2024), a few gradient steps of fine-tuning (Qi et al., 2023c; Zhan…”
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (2024, positive): “In some instances, multimodal attacks have been observed to be stronger than text attacks (Carlini et al., 2023), motivating their inclusion in a standardized evaluation for attacks and defenses.”
Explainability for Large Language Models: A Survey (2023, methodology): “Assistant Model: There are two major limitations of the base models: (1) they cannot follow user instructions as the pre-training data contains few instruction-response examples, and (2) they tend to generate biased and toxic content [Carlini et al. 2023].”
Improving Alignment and Robustness with Circuit Breakers (2024, neutral): “Additionally, multi-modal vision-text attacks range from typographic attacks Goh et al. [18] to gradient-based optimization [9, 59, 6].”
Universal and Transferable Adversarial Attacks on Aligned Language Models (2023, neutral): “However, the success of these attacks on the aligned models that we study was shown to be quite limited [Carlini et al., 2023].”
Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (2023, background): “Carlini et al. (2023) also demonstrate the efficacy of attacks on multi-modal models, where the addition of an adversarial image is able to bypass alignment.”