Gradient-based Adversarial Attacks against Text Transformers

By Chuan Guo, Alexandre Sablayrolles, Herv'e J'egou, Douwe Kiela. 2021

332 citations recorded by the source corpus.

We propose the first general-purpose gradient-based adversarial attack against transformer models. Instead of searching for a single adversarial example, we search for a distribution of adversarial examples parameterized by a continuous-valued matrix, hence enabling gradient-based optimization. We empirically demonstrate that our white-box attack attains state-of-the-art attack performance on a variety of natural language tasks, outperforming prior work in terms of adversarial success rate with matching imperceptibility as per automated and human evaluation. Furthermore, we show that a powerful black-box transfer attack, enabled by sampling from the adversarial distribution, matches or exceeds existing methods, while only requiring hard-label outputs.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Scaling Laws for Reward Model Overoptimization (2022, methodology): “…of adversarial examples in all kinds of neural networks [Szegedy et al., 2013, Lin et al., 2017, Ebrahimi et al., 2018, Dai et al., 2018], and proposed methods to measure and increase neural network robustness [Gu and Rigazio, 2014, Zheng et al., 2016, Carlini et al., 2019, Guo et al., 2021].”
Adversarial Training for High-Stakes Reliability (2022, background): “Adversarial training for language models There is a large and growing literature on both adversarial attacks and adversarial training for large language models [32, 33, 34, 35].”
Security and Privacy Challenges of Large Language Models: A Survey (2024, unknown): “It is imperative to conduct additional comprehensive research and analysis to assess the impacts of these attacks on LLMs and identify effective mitigation strategies.”
Universal and Transferable Adversarial Attacks on Aligned Language Models (2023, unknown): “…highlight the importance of our specific optimizer: previous optimizers, specifically PEZ [Wen et al., 2023] (a gradient-based approach) and GBDA [Guo et al., 2021] (an approach using Gumbel-softmax reparameterization) are not able to achieve any exact output matches, whereas AutoPrompt [Shin et…”
Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (2023, background): “, 2023), GBDA (Guo et al., 2021) and AutoPrompt (Shin et al.”
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (2024, unknown): “These include text optimization methods (Wallace et al., 2019; Guo et al., 2021; Shin et al., 2020; Wen et al., 2023; Jones et al., 2023; Zou et al., 2023), LLM optimizers (Perez et al., 2022; Chao et al., 2023; Mehrotra et al., 2023), and custom jailbreaking templates or pipelines (Liu et al.,…”
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (2024, background): “We accept any jailbreaking attack, including white-box, black-box, universal, transfer, and adaptive attacks, and any jailbreaking defense, all of which are compared using the same set of evaluation metrics.”
Many-shot Jailbreaking (2024, unknown): “Below are additional details on the experimental conditions.”
The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (2024, background): “…et al., 2020; Rafailov et al., 2023) have been developed; however, these methods can be vulnerable to jailbreaks (Wei et al., 2023; Chao et al., 2023; Yao et al., 2023a; Yuan et al., 2023) and adversarial attacks (Wallace et al., 2019; Guo et al., 2021; Jones et al., 2023; Zou et al., 2023b).”
How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs (2024, unknown): “…the exponentially growing user base and the likelihood of increasingly complex persuasive dialogues, we call on the research community to delve deeper into the linguistic patterns and mitigate the potential jailbreak risks arising from the identified factor of humanizing LLMs and human-like…”
Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery (2023, methodology): “We consider two gradient-based methods as baselines: Flu-entPrompt and AutoPrompt (Shi et al., 2022; Shin et al., 2020).”
GPT (Generative Pre-Trained Transformer)— A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions (2023, background): “GPT models are particularly susceptible to adversarial attacks [207].”
Are aligned neural networks adversarially aligned? (2023, methodology): “We evaluate the ARCA attack from Jones et al. [2023] and the GBDA attack from Guo et al. [2021].”
Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts (2021, background): “, “blue” can be perturbed to any other color name) (Guo et al., 2021; Yuan et al., 2021).”