Crafting adversarial input sequences for recurrent neural networks

By Nicolas Papernot, Patrick Drew McDaniel, Ananthram Swami, Richard E. Harang. 2016

495 citations recorded by the source corpus.

Machine learning models are frequently used to solve complex security problems, as well as to make decisions in sensitive situations like guiding autonomous vehicles or predicting financial market behaviors. Previous efforts have shown that numerous machine learning models are vulnerable to adversarial manipulations of their inputs taking the form of adversarial samples. Such inputs are crafted by adding carefully selected perturbations to legitimate inputs so as to force the machine learning model to misbehave, for instance by outputting a wrong class if the machine learning task of interest is classification. In fact, to the best of our knowledge, all previous work on adversarial samples crafting for neural networks considered models used to solve classification tasks, most frequently in computer vision applications. In this paper, we investigate adversarial input sequences for recurrent neural networks processing sequential data. We show that the classes of algorithms introduced previously to craft adversarial samples misclassified by feed-forward neural networks can be adapted to recurrent neural networks. In a experiment, we show that adversaries can craft adversarial sequences misleading both categorical and sequential recurrent neural networks.

Subfields: Artificial Intelligence, Model Architectures, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Universal Adversarial Triggers for Attacking and Analyzing NLP (2019, methodology): “A.1 PGD Replacement Strategy We also consider a token replacement strategy based on projected gradient descent, roughly following Papernot et al. (2016).”
Synthetic and Natural Noise Both Break Neural Machine Translation (2017, background): “…important distinction is often drawn between white-box attacks, where adversarial examples are generated with access to the model parameters, and black-box attacks, where examples are generated without such access (Papernot et al., 2016; 2017; Liu et al., 2017; Narodytska & Kasiviswanathan, 2017).”
ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models (2017, methodology): “s. In addition to image classification, JSMA has been applied to other machine learning tasks such as malware classification [14], and other DNN architectures such as recurrent neural networks (RNNs) [37]. •DeepFool [31]: Inspired from linear classification models and (a) a ZOO black-box targeted at”
MagNet: A Two-Pronged Defense against Adversarial Examples (2017, background): “For example, feedforward convolutional classification networks [2], generative networks [14], and recurrent networks [27].”
Recent Progress on Generative Adversarial Networks (GANs): A Survey (2019, background): “Therefore, each pixel needs to be processed one by one, while GANs directly process the sample in one shot, and this causes GANs to produce a sample faster than PixelRNN.”
One Pixel Attack for Fooling Deep Neural Networks (2017, background): “perturbation can be also possible in other domains, such as speech recognition [33], natural language processing [34] and malware classification [35].”
HotFlip: White-Box Adversarial Examples for Text Classification (2017, background): “Similar small drops in human performance have been reported for image classification (Papernot et al., 2016) and text comprehension (Jia and Liang, 2017).”
Pathologies of Neural Models Make Interpretations Difficult (2018, background): “Other ways to expose these types of examples in NLP have taken the form of word and character-level modifications (Papernot et al., 2016; Ebrahimi et al., 2017; Belinkov and Bisk, 2017).”
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, neutral): “[110] successfully generated adversarial input sequences for Recurrent Neural Networks (RNNs).”
DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars (2017, background): “These types of attacks are known to affect a broad spectrum of tasks such as image recognition [36, 39, 51, 54, 61, 62, 64, 65, 77], face detection/verification [74, 80], malware detection [27, 41, 53, 84], and text analysis [58, 66].”
Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems (2017, background): “Adversarial machine learning is a popular research area that focuses on generating errorinducing test inputs for ML models by adding minimal perturbations to an existing input [16], [26], [27], [30], [35], [41], [44], [51], [53] and studying how to improve the robustness of the ML systems against su”
Negative Training for Neural Dialogue Response Generation (2019, background): “sentence classification, or seq2seq models) (Papernot et al., 2016; Samanta and Mehta, 2017; Liang et al., 2018; Ebrahimi et al., 2017; Belinkov and Bisk, 2017; Chen et al., 2017).”
Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples (2018, methodology): “[33] first uses Fast Gradient Sign Method (FGSM) to conduct an attack on RNN/LSTM-based classification problems.”
Detecting egregious responses in neural sequence-to-sequence models (2018, background): “ut space, there’s a recent and growing interest in analyzing the robustness of deep learning models for NLP tasks. Most of work focuses on sentence classiﬁcation tasks (e.g. sentiment classiﬁcation) (Papernot et al., 2016; Samanta & Mehta, 2017; Liang et al., 2018; Ebrahimi et al., 2017), and so”
Universal Adversarial Triggers for NLP (2019, methodology): “A.1 PGD Replacement Strategy We also consider a token replacement strategy based on projected gradient descent, roughly following Papernot et al. (2016).”
Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder (2020, background): “, 2018), editing the words and characters directly (Papernot et al., 2016; Alzantot et al., 2018; Ebrahimi et al., 2017) or paraphrasing sentences (Iyyer et al.”
Can you fool AI with adversarial examples on a visual Turing test? (2017, background): “While most previous works focus on image classification models, a new line of research studies adversarial examples against deep neural networks for other tasks, such as recurrent neural networks for text processing [43], [20], deep reinforcement learning models for game playing [29], [19], [24], se”
Weight Poisoning Attacks on Pretrained Models (2020, background): “…attacks, ﬁrst investigated by Szegedy et al. (2013) and Goodfellow et al. (2015) in computer vision and later extended to text classiﬁcation (Papernot et al., 2016; Ebrahimi et al., 2018b; Li et al., 2018; Hos-seini et al., 2017) and translation (Ebrahimi et al., 2018a; Michel et al.,…”