Adversarial Training Methods for Semi-Supervised Text Classification

By Takeru Miyato, Andrew M. Dai, Ian J. Goodfellow. 2016

1,155 citations recorded by the source corpus.

Adversarial training provides a means of regularizing supervised learning algorithms while virtual adversarial training is able to extend supervised learning algorithms to the semi-supervised setting. However, both methods require making small perturbations to numerous entries of the input vector, which is inappropriate for sparse high-dimensional inputs such as one-hot word representations. We extend adversarial and virtual adversarial training to the text domain by applying perturbations to the word embeddings in a recurrent neural network rather than to the original input itself. The proposed method achieves state of the art results on multiple benchmark semi-supervised and purely supervised tasks. We provide visualizations and analysis showing that the learned word embeddings have improved in quality and that while training, the model is less prone to overfitting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “Otherwise, the implementation is as described in (Miyato et al., 2017a); we based our implementation off of their code 4 .”
Universal Language Model Fine-tuning for Text Classification (2018, neutral): “(Johnson and Zhang, 2016) 5.9 TBCNN (Mou et al., 2015) 4.0 Virtual (Miyato et al., 2016) 5.9 LSTM-CNN (Zhou et al., 2016) Johnson and Zhang (2017).”
Learned in Translation: Contextualized Word Vectors (2017, neutral): “…87.3 bmLSTM [Radford et al., 2017] 92.9 re-read LSTM [Sha et al., 2016] 87.5 TRNN [Dieng et al., 2016] 93.8 btree-LSTM [Paria et al., 2016] 87.6 oh-LSTM [Johnson and Zhang, 2016] 94.1 600D ESIM [Chen et al., 2016] 88.0 Virtual [Miyato et al., 2017] 94.1 BCN+Char+CoVe (Ours) 88.1 Test Performance.”
XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, methodology): “Following previous work on text classification [40, 20], we evaluate XLNet on the following benchmarks: IMDB, Yelp-2, Yelp-5, DBpedia, AG, Amazon-2, and Amazon-5.”
Technical report on Conversational Question Answering (2019, methodology): “Assuming that vw is the embedding vector of word w and θ̂ represents the current parameters of the model, the adversarial embedding vector v∗ w (Miyato et al., 2016) is:”
Unsupervised Data Augmentation for Consistency Training (2019, background): “Other works in the consistency training family mostly differ in how the noise is defined: Pseudoensemble [2] directly applies Gaussian noise and Dropout noise; VAT [6, 43] defines the noise by approximating the direction of change in the input space that the model is most sensitive to; Cross-view tr”
Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function (2019, methodology): “In this work, we make adversarial perturbations to the input word embeddings Miyato, Dai, and Goodfellow 2016).”
Making Pre-trained Language Models Better Few-shot Learners (2021, background): “learning (Miyato et al., 2017; Xie et al., 2020; Chen et al., 2020), where a set of unlabeled examples are given; (2) meta-learning (Yu et al.”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “There are also some parallel studies on applying AT to language modeling [67] and natural language understanding [45, 85, 24].”
Revisiting Self-Training for Neural Sequence Generation (2019, neutral): “These approaches have been very successful on classiﬁcation tasks (Rasmus et al., 2015; Miyato et al., 2017; Laine & Aila, 2017; Miyato et al., 2018; Xie et al., 2019) given that a reasonable amount of predictions of unlabeled data (at least the ones with high conﬁdence) are correct, but their…”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “More broadly, semi-supervised learning additionally leverages unlabeled task-speciﬁc data, where data augmentation [7, 73], regularization [43] and bootstrapping [56] methods are commonly used.”
Learning to Generate Reviews and Discovering Sentiment (2017, methodology): “One exception is that we use an L1 penalty for text classiﬁcation results instead of L2 as we found this performed better in the very low data regime.”
Towards Open Intent Discovery for Conversational Text (2019, methodology): “Since adversarial training considers continuous perturbations to inputs, we add adversarial noise at the embedding layer, as in [37].”
HotFlip: White-Box Adversarial Examples for Text Classification (2017, methodology): “Misc. error Success rate Baseline 8.27% 98.16% Adv-tr (Miyato et al., 2017) 8.03% 87.43% Adv-tr (black-box) 8.60% 95.63% Adv-tr (white-box) 7.65 % 69.32 % mainly because the adversary that we use at test time, which uses beam search, is strictly stronger than our model’s internal adversary.”
MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification (2020, methodology): “Miyato et al. (2017) utilized adversarial and virtual adversarial training to the text domain by applying perturbations to the word embeddings.”
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, neutral): “[113]”
FreeLB: Enhanced Adversarial Training for Natural Language Understanding (2019, background): “Miyato et al. (2017) extend adversarial and virtual adversarial training (Miyato et al., 2019) to the text domain to improve the performance on semi-supervised classiﬁcation tasks.”
TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency (2016, background): “This is close to the state-of-the-art 5.91% (Miyato et al., 2016) despite that we do not use the labels and adversarial training in the feature extraction stage.”
Deconvolutional Paragraph Representation Learning (2017, methodology): “Typically, RNN-based methods consider either i) training a sequence-to-sequence RNN autoencoder, or a RNN classifier that is robust to adversarial perturbation, as initialization for the encoder in the supervised model [1, 4]; or, ii) learning latent representation via a sequence-to-sequence RNN aut”