Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning

By Mehdi S. M. Sajjadi, Mehran Javanmardi, Tolga Tasdizen. 2016

1,270 citations recorded by the source corpus.

Effective convolutional neural networks are trained on large sets of labeled data. However, creating large labeled datasets is a very costly and time-consuming task. Semi-supervised learning uses unlabeled data to train a model with higher accuracy when there is a limited set of labeled data available. In this paper, we consider the problem of semi-supervised learning with convolutional neural networks. Techniques such as randomized data augmentation, dropout and random max-pooling provide better generalization and stability for classifiers that are trained using gradient descent. Multiple passes of an individual sample through the network might lead to different predictions due to the non-deterministic behavior of these techniques. We propose an unsupervised loss function that takes advantage of the stochastic nature of these methods and minimizes the difference between the predictions of multiple passes of a training sample through the network. We evaluate the proposed method on several benchmark datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, background): “Recent research on computer vision addresses this by adding noise to the student’s input, training the model so it is robust to input perturbations (Sajjadi et al., 2016; Wei et al., 2018).”
Unsupervised Data Augmentation for Consistency Training (2019, background): “Existing works in consistency training does make use of data augmentation [4, 7]; however, they only apply weak augmentation methods such as random translations and cropping.”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, methodology): “ for a survey). An increasingly popular approach in deep learning consists in propagating information learned from labeled images to unlabeled ones [23,39] while enforcing invariance to perturbations [44,59,65]. In relying on labeled data to provide a learning signal for unlabeled data, these method”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, methodology): “Another family of methods are based on label consistency regularization [60–62, 39, 14, 13, 63, 15], where unlabeled examples are directly used as a regularizer to encourage task prediction consistency.”
Training Generative Adversarial Networks with Limited Data (2020, background): “Consistency regularization states that two sets of augmentations, applied to the same input image, should yield the same output [35, 27].”
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020, methodology): “This has led to a plethora of SSL methods that are designed for deep networks [28, 39, 21, 43, 3, 45, 2, 22, 38].”
MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, methodology): “As baselines, we consider the four methods considered in [35] (Π-Model [25, 40], Mean Teacher [44], Virtual Adversarial Training [31], and Pseudo-Label [28]) which are described in section 2.”
Learning deep representations by mutual information estimation and maximization (2018, methodology): “…approaches include deep volume-preserving maps (Dinh et al., 2014, 2016), deep clustering (Xie et al., 2016; Chang et al., 2017), noise as targets (NAT, Bojanowski & Joulin, 2017), self-supervised methods (Doersch & Zisserman, 2017), and co-learning (Dosovitskiy et al., 2016; Sajjadi et al., 2016).”
AutoAugment: Learning Augmentation Policies from Data (2018, methodology): “icy to train models on reduced CIFAR-10 (the same 4,000 example training set that we use to ﬁnd the best policy). Similar to the experimental convention used by the semi-supervised learning community [60,40,51,31,44] we train on 4,000 labeled samples. But we do not use the 46,000 unlabeled samples d”
Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning (2017, methodology): “This cost was introduced by [15], and similar idea has been used in [35].”
Temporal Ensembling for Semi-Supervised Learning (2016, methodology): “The recently introduced transform/stability loss of Sajjadi et al. (2016b) is based on the same principle as our work, and the Π -model can be seen as a special case of it.”
Colorization as a Proxy Task for Visual Understanding (2017, background): “Semi-supervised learning, jointly training a supervised and an unsupervised loss, offers a middle ground [7, 34].”
A soft nearest-neighbor framework for continual semi-supervised learning (2022, neutral): “Notable examples of this consistency based category are [4, 5, 27,33,35,43,49,52,60].”
Continual semi-supervised learning through contrastive interpolation consistency (2021, background): “ers [37, 9] or the predictions on different augmentations of the same data-point by the same learner [8, 38, 39, 40].”
Learning to Compose Domain-Specific Transformations for Data Augmentation (2017, methodology): “Additionally, we incorporate a basic transformation regularization term as in [24] (see Supplemental Materials), and train for the last ten epochs without applying any transformations as in [11].”
There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average (2018, methodology): “Π Model The Π model, introduced in Laine and Aila (2017) and Sajjadi et al. (2016), uses the student model f as its own teacher.”
Data Distillation: Towards Omni-Supervised Learning (2017, background): “[35] enforce consistency between outputs computed for different transformations of input examples.”