Distributional Smoothing with Virtual Adversarial Training

By Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, Shin Ishii. 2015

467 citations recorded by the source corpus.

We propose local distributional smoothness (LDS), a new notion of smoothness for statistical model that can be used as a regularization term to promote the smoothness of the model distribution. We named the LDS based regularization as virtual adversarial training (VAT). The LDS of a model at an input datapoint is defined as the KL-divergence based robustness of the model distribution against local perturbation around the datapoint. VAT resembles adversarial training, but distinguishes itself in that it determines the adversarial direction from the model distribution alone without using the label information, making it applicable to semi-supervised learning. The computational cost for VAT is relatively low. For neural network, the approximated gradient of the LDS can be computed with no more than three pairs of forward and back propagations. When we applied our technique to supervised and semi-supervised learning for the MNIST dataset, it outperformed all the training methods other than the current state of the art method, which is based on a highly advanced generative model. We also applied our method to SVHN and NORB, and confirmed our method's superior performance over the current state of the art semi-supervised method applied to these datasets.

Subfields: Aerospace Engineering, Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, background): “Therefore we see much better results from VAT on semi-supervised CIFAR-10 compared to on our NLP tasks.”
Adversarial Training Methods for Semi-Supervised Text Classification (2016, methodology): “Virtual adversarial training (Miyato et al., 2016) extends the idea of adversarial training to the semi-supervised regime and unlabeled examples.”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, neutral): “14 ) VAT [32] 2 .”
A Closer Look at Memorization in Deep Networks (2017, background): “(Miyato et al., 2015) deﬁne virtual adversarial examples via changes in the predictive distribution instead, thus extending the deﬁ-nition to unlabeled data-points.”
Adversarial Autoencoders (2015, methodology): “On the MNIST dataset with 100 and 1000 labels, the performance of AAEs is signiﬁcantly better than VAEs, on par with VAT [Miyato et al., 2015] and CatGAN [Springenberg, 2015], but is outperformed by the Ladder networks [Rasmus et al., 2015] and the ADGM [Maaløe et al., 2016].”
Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning (2017, background): “This article also extends our earlier work [27] in four aspects: • clariﬁcation of the objective function”
Semi-supervised Learning with Ladder Networks (2015, background): “Recently Miyato et al. (2015) achieved impressive results with a regularization method that is similar to the idea of contractive cost.”
Auxiliary Deep Generative Models (2016, methodology): “We normalized all NORB images following Miyato et al. (2015) using image pairs of 32 x 32 resulting in a vectorized input of dim( x ) = 2048 .”
Temporal Ensembling for Semi-Supervised Learning (2016, neutral): “10 Virtual Adversarial (Miyato et al., 2016) 24 .”
Regularizing Neural Networks by Penalizing Confident Output Distributions (2017, background): “Virtual adversarial training (VAT) (Miyato et al., 2015) is another promising smoothing regularizer.”
Adversarial Machine Learning at Scale (2016, methodology): “This differs from results of adversarial training reported previously, where adversarial training increased accuracy on the test set (Goodfellow et al., 2014; Miyato et al., 2016b;a).”
Certified Defenses against Adversarial Examples (2018, background): “There are also other attacks based on, e.g., saliency maps (Papernot et al., 2016a), KL divergence (Miyato et al., 2015), and elastic net optimization (Chen et al., 2017); many of these attacks are collated in the cleverhans repository (Goodfellow et al., 2016).”
Certifying Some Distributional Robustness with Principled Adversarial Training (2017, background): “, 2007; Delage & Ye, 2010; Goh & Sim, 2010), as well as non-parametric distances for probability measures such as f -divergences (Ben-Tal et al., 2013; Bertsimas et al., 2013; Lam & Zhou, 2015; Miyato et al., 2015; Duchi et al., 2016; Namkoong & Duchi, 2016), and Wasserstein distances (Esfahani & Ku”
Parseval Networks: Improving Robustness to Adversarial Examples (2017, methodology): “However, the bulk of recent proposals relies on data augmentation (Goodfellow et al., 2015; Miyato et al., 2015; Moosavi-Dezfooli et al., 2015; Shaham et al., 2015; Szegedy et al., 2014; Kurakin et al., 2016).”
SafetyNet: Detecting and Rejecting Adversarial Examples Robustly (2017, background): “One way to build a network that is robust to adversarial examples is to train networks with enhanced training data (adding adversarial samples [18]); this approach faces difficulties, because the dimension of the images and features in networks means an unreasonable quantity of training data is requ”
Improving the Robustness of Deep Neural Networks via Stability Training (2016, background): “Previous work has shown that training a classifier to resist adversarial perturbation can improve its performance on both the original data and on perturbed data [1, 6].”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, background): “Also related is virtual adversarial training (Miyato et al., 2015), which includes examples that are close to the current input but cause a maximal change in the model output, and mixup (Zhang et al.”
Measuring Neural Net Robustness with Constraints (2016, background): “In practice, X = Xtest is taken to be the test set.”