Adversarial Dropout for Supervised and Semi-supervised Learning

By Sungrae Park, Jun-Keon Park, Su-Jin Shin, Il-Chul Moon. 2017

186 citations recorded by the source corpus.

Recently, training with adversarial examples, which are generated by adding a small but worst-case perturbation on input examples, has improved the generalization performance of neural networks. In contrast to the biased individual inputs to enhance the generality, this paper introduces adversarial dropout, which is a minimal set of dropouts that maximize the divergence between 1) the training supervision and 2) the outputs from the network with the dropouts. The identified adversarial dropouts are used to automatically reconfigure the neural network in the training process, and we demonstrated that the simultaneous training on the original and the reconfigured network improves the generalization performance of supervised and semi-supervised learning tasks on MNIST, SVHN, and CIFAR-10. We analyzed the trained model to find the performance improvement reasons. We found that adversarial dropout increases the sparsity of neural networks more than the standard dropout. Finally, we also proved that adversarial dropout is a regularization term with a rank-valued hyper-parameter that is different from a continuous-valued parameter to specify the strength of the regularization.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Self-Training With Noisy Student Improves ImageNet Classification (2019, background): “Apart from self-training, another important line of work in semi-supervised learning [12, 101] is based on consistency training [5, 64, 47, 83, 56, 52, 62, 13, 16, 60, 2, 49, 86, 89, 8, 96, 46, 7].”
A soft nearest-neighbor framework for continual semi-supervised learning (2022, background): “Another class of approaches was derived with the idea of imposing similar predictions from the network for two samples obtained with different input perturbations [3,4,27,33, 35, 41, 47, 50, 60].”
There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average (2018, background): “Currently the best semi-supervised results are obtained by consistency-enforcing approaches (Bachman et al., 2014; Laine and Aila, 2017; Tarvainen and Valpola, 2017; Miyato et al., 2017; Park et al., 2017).”
A Fourier-based Framework for Domain Generalization (2021, methodology): “[34] develop different techniques by replacing the stochastic perturbations with adversarial ones.”
A Survey on Deep Semi-Supervised Learning (2021, methodology): “(VAdD) [29] also employs adversarial training in addition to the P Model.”
An Incremental Self-Labeling Strategy for Semi-Supervised Deep Learning Based on Generative Adversarial Networks (2020, background): “Other than learning a perturbed noise, the adversarial dropout [14] alternatively learns dropout masks, the directions of which are optimized adversarially to the model’s virtual label assignment. fast-SWA (fast Stochastic Weight Averaging) [13] is proposed to further stabilize the training process…”
Self-Challenging Improves Cross-Domain Generalization (2020, background): “(c) AdversarialDropout [21,12].”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, methodology): “(3), stochastic perturbation [52], [53], [54] and ensembling [37], [39], [86] are proposed.”
Adversarial Personalized Ranking for Recommendation (2018, methodology): “Thus, we employ the fast gradient method proposed in Goodfellow et al. [15], a common choice in adversarial training [24, 26, 34].”
Guided Collaborative Training for Pixel-wise Semi-Supervised Learning (2020, background): “Apart from the perturbations in the inputs, Dropout perturbs the predictions through a random selection of nodes [34].”
A survey on semi-supervised learning (2019, background): “Park et al. (2018) reported small performance improvements over virtual adversarial training and the -model.”
Improving Consistency-Based Semi-Supervised Learning with Weight Averaging (2018, background): “The previous best numbers are obtained from [28](1), [21](2), [15](3).”
An Overview of Deep Semi-Supervised Learning (2020, methodology): “[117] is limited to fully-connected networks, to use AdD in a wider range of tasks, Lee et al .”