S4L: Self-Supervised Semi-Supervised Learning

By Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, Lucas Beyer. 2019

864 citations recorded by the source corpus.

This work tackles the problem of semi-supervised learning of image classifiers. Our main insight is that the field of semi-supervised learning can benefit from the quickly advancing field of self-supervised visual representation learning. Unifying these two approaches, we propose the framework of self-supervised semi-supervised learning (S4L) and use it to derive two novel semi-supervised image classification methods. We demonstrate the effectiveness of these methods in comparison to both carefully tuned baselines, and existing semi-supervised learning methods. We then show that S4L and existing semi-supervised methods can be jointly trained, yielding a new state-of-the-art result on semi-supervised ILSVRC-2012 with 10% of labels.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, methodology): “Figure 2 decomposes the VTAB tasks into their respective groups, and compares to previous SOTA methods on this benchmark: BiT, VIVI – a ResNet co-trained on ImageNet and Youtube (Tschannen et al., 2020), and S4L – supervised plus semi-supervised learning on ImageNet (Zhai et al., 2019a).”
Unsupervised Data Augmentation for Consistency Training (2019, background): “In parallel to our work, S4L [40] and CPC [41] also show significant improvements on ImageNet.”
On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “We include self-supervised SimCLR [6] (variants: linear classifier on fixed representation (lin), fine-tuned on 10% (ft-10), and 100% (ft-100) of the IMAGENET data), and self-supervised-semi-supervised (S4L) [71] models that have been fine-tuned to 10% and 100% of the IMAGENET data.”
Are we done with ImageNet? (2020, methodology): “We start by assembling a set of 19 models [3, 27, 4, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37] which vary along a number of dimensions, including architecture, training objective and usage of external data (see appendix A).”
Big Transfer (BiT): General Visual Representation Learning (2019, background): “Recent works combine self-supervised and semi-supervised learning to attain good performance with fewer labels on ImageNet [57,12].”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “For models co-trained with a supervised loss based on a fraction of ImageNet we additionally use the same HSVspace color randomization as [68].”
Self-Training With Noisy Student Improves ImageNet Classification (2019, neutral): “in a video, which is not as general as our work. Semi-supervised Learning. Apart from self-training, another important line of work in semi-supervised learning [9,85] is based on consistency training [6,4,53,36, 70,45,41,51,10,12,49,2,38,72,74,5,81]. These works constrain model predictions to be inv”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “The second class of methods attempts to propagate the knowledge extracted from the subset of labeled examples to unlabeled examples while being invariant to augmentation or other perturbations[68, 65].”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, methodology): “-speciﬁc performance, or with a smaller model architecture, which leads to a compact model. 3 Empirical Study 3.1 Settings and Implementation Details Following the semi-supervised learning setting in [30, 19, 1], we evaluate the proposed method on ImageNet ILSVRC-2012 [21]. While all ˘1.28 million i”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “Semi-supervised learning on ImageNet We follow the semi-supervised learning protocol of [8, 77].”
VirTex: Learning Visual Representations from Textual Annotations (2020, background): “S4L [44] uses self-supervised learning for unlabeled images, along with supervised learning for labeled images.”
Grafit: Learning fine-grained image representations with coarse labels (2020, background): “Recent works on semi-supervised learning [4, 5, 48, 62, 66, 69] rely on both supervised and self-supervised losses to get information from unlabelled data.”
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020, background): “While S(4)L [57] holds state-of-the-art on semi-supervised ImageNet with a 26.”
Self-Supervised Learning of Pretext-Invariant Representations (2019, methodology): “PIRL performs at least as well as S4L [84] and better than VAT [22], which are both methods specifically designed for semi-supervised learning.”
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, background): “Another big theme in semi-supervised learning is consistency [34–36].”
Improved Noisy Student Training for Automatic Speech Recognition (2020, methodology): “Another direction of semi-supervised learning research is to use consistency-based methods (e.g., [21, 22, 23]).”
Scaling Vision Transformers (2021, background): “Ten images per class is less than 1% of ImageNet data (13 examples per class), as commonly used in self-supervised and semisupervised learning [51].”
What makes for good views for contrastive learning (2020, background): “Leveraging labeled data in contrastive representation learning has been shown to guide representations towards task-relevant features that improve performance [61, 25, 29, 57].”
Prototypical Contrastive Learning of Unsupervised Representations (2020, neutral): “4 SL Exemplar [40] ResNet-50v2 47.”