Distributional Smoothing by Virtual Adversarial Examples

By Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, Shin Ishii. 2015

73 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Generative ConvNets via Multi-grid Modeling and Sampling (2017, methodology): “Our method also outperforms some meth-ods that are speciﬁcally designed for semi-supervised learning, such as DGN [17], virtual adversarial [30] and auxiliary deep generative model [29].”
Improved Techniques for Training GANs (2016, methodology): “DGN[21] 333 ± 14 VirtualAdversarial[22] 212 CatGAN[14] 191 ± 10 SkipDeepGenerativeModel[23] 132 ± 7 Laddernetwork[24] 106 ± 37 AuxiliaryDeepGenerativeModel[23] 96 When presented with 50% real and 50% fake data generated by our best CIFAR-10 model, MTurk users correctly categorized 78 .”
Semi-supervised Learning with Ladder Networks (2015, methodology): “Embedding (Weston et al. , 2012) 16.86 5.73 1.5 Transductive SVM (from Weston et al. , 2012) 16.81 5.38 1.40* MTC (Rifai et al. , 2011b) 12.03 3.64 0.81 Pseudo-label (Lee, 2013) 10.49 3.46 AtlasRBF (Pitelis et al. , 2014) 8.10 ( ± 0.95) 3.68 ( ± 0.12) 1.31 DGN (Kingma et al. , 2014) 3.33 ( 0.14) 2.4”
Regularizing Neural Networks by Penalizing Confident Output Distributions (2017, background): “Virtual adversarial training (VAT) (Miyato et al., 2015) is another promising smoothing regularizer.”
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks (2015, methodology): “[29] introduced another sensitivity estimation measure, named the Local Distribution Smoothness, based on the Kullback-Leibler divergence, a measure of the difference between two probability distributions.”
Learning with a Strong Adversary (2015, background): “Moreover, the one proposed in Miyato et al. (2015) can be interpreated as D KL ( y i (cid:107) p )+ D KL ( p (cid:107) ˜ p ) .”
Deep Bayesian Active Learning with Image Data (2017, neutral): “…2012) 5.38% MTC (Rifai et al., 2011) 3.64% Pseudo-label (Lee, 2013) 3.46% AtlasRBF (Pitelis et al., 2014) 3.68% DGN (Kingma et al., 2014) 2.40% Virtual Adversarial (Miyato et al., 2015) 1.32% Ladder Network ( Γ -model) (Rasmus et al., 2015) 1.53% Ladder Network (full) (Rasmus et al., 2015) 0.84%”
Label Refinery: Improving ImageNet Classification through Label Progression (2018, background): “[13] introduces a notion of local distributional smoothness in model outputs based on the smoothness of the model’s outputs when inputs are perturbed.”
Few-shot learning of neural networks from scratch by pseudo example optimization (2018, methodology): “In particular for the use of Gaussian processes as referencd models, they introduce local smoothness of predictions into neural networks, which is lacked in standard neural networks [3] and is proved to be effective for overfitting [23].”
Geometry-aware Instance-reweighted Adversarial Training (2020, methodology): “Note that TRADES utilizes virtual adversarial data (Miyato et al., 2016) for updating the current model.”
Attacks Which Do Not Kill Training Make Adversarial Learning Stronger (2020, background): “Similar to virtual adversarial training (VAT) adding a regularization term to the loss function [41], which regularizes the output distribution by its local sensitivity of the output w.r.t. input, the objective of TRADES is where β > 0 is a regularization parameter, which controls the trade-oﬀ…”
Cascade Adversarial Machine Learning Regularized with a Unified Embedding (2017, unknown): “We also observe increased accuracy for the clean images for some cases compared to the baseline standard training as in [3, 16], and the authors in [5] explains this might be because the adversarial training acts as a regularizer for overﬁtted model.”
Adversarial Images for Variational Autoencoders (2016, background): “Training models to resist adversarial attacks was advanced as a form of regularization [3, 5].”
Imitation networks: Few-shot learning of neural networks from scratch (2018, methodology): “ussian processes as referencd models, they introduce local smoothness of predictions into neural networks, which is lacked in standard neural networks [3] and is proved to be effective for overﬁtting [23]. 3.3 Fidelity weighting We note that all the pseudo training examples are not always useful for”
Deep Growing Learning (2017, methodology): “The second group of comparing methods includes thirteen representative semi-supervised deep learning models: Neural Networks (NN) [29], Neural Networks with Embedding regularizer (EmbedNN) [33], Convolutional Neural Network (CNN) [29], Convolutional Neural Network for Embedding (EmbedCNN) [33], Cont”
Fine-grained Recognition in the Noisy Wild: Sensitivity Analysis of Convolutional Neural Networks Approaches (2016, methodology): “An implicit data augmentation technique is given in [16], where an additional regularization term is used to minimize the Kullback Leibler divergence between the original and the adversarial posterior distribution.”