Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning

By Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Shin Ishii. 2017

3,088 citations recorded by the source corpus.

We propose a new regularization method based on virtual adversarial loss: a new measure of local smoothness of the conditional label distribution given input. Virtual adversarial loss is defined as the robustness of the conditional label distribution around each input data point against local perturbation. Unlike adversarial training, our method defines the adversarial direction without label information and is hence applicable to semi-supervised learning. Because the directions in which we smooth the model are only “virtually” adversarial, we call our method virtual adversarial training (VAT). The computational cost of VAT is relatively low. For neural networks, the approximated gradient of virtual adversarial loss can be computed with no more than two pairs of forward- and back-propagations. In our experiments, we applied VAT to supervised and semi-supervised learning tasks on multiple benchmark datasets. With a simple enhancement of the algorithm based on the entropy minimization principle, our VAT achieves state-of-the-art performance for semi-supervised learning tasks on SVHN and CIFAR-10.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “Otherwise, the implementation is as described in (Miyato et al., 2017a); we based our implementation off of their code 4 .”
Adversarial Training for Large Neural Language Models (2020, background): “…Zhu et al. (2019) and most other approaches, we follow Jiang et al. (2019) to regularize the standard objective using virtual adversarial training (Miyato et al., 2018): Effectively, the adversarial term favors label smoothness in the embedding neighborhood, and α is a hyperparameter that…”
Unsupervised Data Augmentation for Consistency Training (2019, methodology): “(1) where CE denotes cross entropy, q(x̂ | x) is a data augmentation transformation and ✓̃ is a fixed copy of the current parameters ✓ indicating that the gradient is not propagated through ✓̃, as suggested by VAT [6].”
Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function (2019, background): “Entropy Minimization (EM) In addition to supervised cross-entropy loss, we also minimize the conditional entropy of the estimated class probabilities (Grandvalet and Bengio 2004; Miyato et al. 2018).”
Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “Apart from self-training, another important line of work in semi-supervised learning [9, 85] is based on consistency training [6, 4, 53, 36, 70, 45, 41, 51, 10, 12, 49, 2, 38, 72, 74, 5, 81].”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, neutral): “, 2019) or adversarial perturbation (Miyato et al., 2018).”
S4L: Self-Supervised Semi-Supervised Learning (2019, methodology): “Semi-supervised learning algorithms are typically [28, 22, 2, 39, 1, 21] evaluated on small-scale datasets such as CIFAR-10 [17] and SVHN [23].”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, neutral): “[37, 38, 30] ResNet-50 - - 47.0 83.4 UDA (w. RandAug) [14] ResNet-50 - 68.8 - 88.5 FixMatch (w. RandAug) [15] ResNet-50 - 71.5 - 89.1 S4L (Rot+VAT+Entropy Min.)”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “In the semi-supervised setting [55, 56], an unsupervised loss is combined with a classification loss over a handful of labels to ground the training [19, 20, 57, 58, 59, 60, 61, 62].”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “ates that the conﬁdence level of the prediction, characterized by the probability vector over the simplex n(nis the number of classes), should also be close. Similar techniques are used in Virtual AT [46], TRADES [84], and UDA [75]. However, previous work mostly focuses on semi-supervised learning o”
Flow Contrastive Estimation of Energy-Based Models (2019, methodology): “Following the setting in [49], we use two types of CNN structures (‘Conv-small’and ‘Conv-large’) for EBMs, which are commonly used in state-of-the-art semi-supervised learning methods.”
Self-Supervised Learning of Pretext-Invariant Representations (2019, neutral): “[22, 52] R-50v2 47.”
MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, methodology): ““Virtual Adversarial Training” [30] (VAT) addresses this by instead computing an additive perturbation to apply to the input which maximally changes the output class distribution.”
Revisiting Self-Training for Neural Sequence Generation (2019, neutral): “These approaches have been very successful on classiﬁcation tasks (Rasmus et al., 2015; Miyato et al., 2017; Laine & Aila, 2017; Miyato et al., 2018; Xie et al., 2019) given that a reasonable amount of predictions of unlabeled data (at least the ones with high conﬁdence) are correct, but their…”
AutoAugment: Learning Augmentation Policies from Data (2018, neutral): “AutoAugment results on the reduced set are again comparable to the leading semi-supervised methods, which range from 5.42% to 3.86% [63].”
Prototypical Contrastive Learning of Unsupervised Representations (2020, neutral): “[40, 39] ResNet-50v2 - 47.0 83.4 S 4 L Exemplar [39] ResNet-50v2 - 47.0 83.7 S 4 L Rotation [39] ResNet-50v2 - 53.4 83.8 Self-supervised learning methods: Instance Discrimination [1] ResNet-50 200 39.2 77.4 Jigsaw [24, 4] ResNet-50 90 45.3 79.3 SimCLR [8] ResNet-50-MLP 200 56.5 82.7 MoCo [3]…”
Scalable Private Learning with PATE (2018, methodology): “…PATE’s flexibility and apply the technique that performs best on each dataset: semi-supervised learning with Generative Adversarial Networks (Salimans et al., 2016) for MNIST and SVHN, Virtual Adversarial Training (Miyato et al., 2017) for Glyph, and fully-supervised random forests for UCI Adult.”