Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey

By Longlong Jing, Yingli Tian. 2019

2,029 citations recorded by the source corpus.

Large-scale labeled data are generally required to train deep neural networks in order to obtain better performance in visual feature learning from images or videos for computer vision applications. To avoid extensive cost of collecting and annotating large-scale datasets, as a subset of unsupervised learning methods, self-supervised learning methods are proposed to learn general image and video features from large-scale unlabeled data without using any human-annotated labels. This paper provides an extensive review of deep learning-based self-supervised general visual feature learning methods from images or videos. First, the motivation, general pipeline, and terminologies of this field are described. Then the common deep neural network architectures that used for self-supervised learning are summarized. Next, the schema and evaluation metrics of self-supervised learning methods are reviewed followed by the commonly used datasets for images, videos, audios, and 3D data, as well as the existing self-supervised visual feature learning methods. Finally, quantitative performance comparisons of the reviewed methods on benchmark datasets are summarized and discussed for both image and video feature learning. At last, this paper is concluded and lists a set of promising future directions for self-supervised visual feature learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “C V ] 22 M a y 2019 sentations that are capable of predicting pixel-level details [34, 54, 12], and self-supervised methods that formulate alternative prediction tasks in which missing information is imputed from other parts of the image [33].”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “[27] for an exhaustive and detailed review of this literature.”
Class-Incremental Learning: Survey and Performance Evaluation on Image Classification (2020, background): “learning [118] is only little explored within the context of IL, and is expected to gain interest.”
Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer (2021, neutral): “Other approaches for leveraging unlabeled data is to use it for an auxiliary loss such as generative loss [16, 20] or self-supervised learning [59].”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, background): “For an extended review on self-supervised learning in the visual domain see [23].”
Image Segmentation Using Deep Learning: A Survey (2020, background): “few shot) learning [186] and unsupervised learning [187] are becoming very active research areas.”
How Well Do Self-Supervised Models Transfer? (2020, background): “Self-supervised learning Self-supervised representation learning is now a large topic that it is impossible to cover completely here, and we point the reader to excellent recent surveys [26, 37] for thorough reviews.”
Conterfactual Generative Zero-Shot Semantic Segmentation (2021, methodology): “We generally use three groups of methods to train unsupervised representation: transformation-equivariant representations [9], self-supervised methods [10] and generative models [11].”
SelFlow: Self-Supervised Learning of Optical Flow (2019, background): “It is widely used for learning feature representations from unlabeled data [21].”
Global and Local Contrastive Self-Supervised Learning for Semantic Segmentation of HR Remote Sensing Images (2021, methodology): “Self-supervised learning (SSL) [34]–[36] provides a new paradigm, as shown in Fig.”
Contrastive Learning With Stronger Augmentations (2021, background): “For more details about these works, please refer to the survey by [36].”
Remote Sensing Image Scene Classification With Self-Supervised Paradigm Under Limited Labeled Samples (2020, background): “Most recently, a new trend is observed in machine learning, which is learning representations by self-supervised learning (SSL) methods without any additional annotation cost [8].”
Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals (2021, positive): “ckward reasoning has been seldom considered in multi-hop KBQA task, especially its correspondence with forward reasoning. Such an idea is indeed related to recent progress in self-supervised learning [15], in which we leverage internal supervision signal to learn the model. 5 EXPERIMENT In this sect”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “Our survey is related to other surveys on semi-supervised learning [8], [13], [17], self-supervised learning [18], or both topics [19].”
Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis (2021, methodology): “The auto-encoding approach we employ is a typical method for self-supervised learning, which has been well recognized to improve the model robustness and generalization ability (He et al., 2020; Hendrycks et al., 2019; Jing & Tian, 2020; Goyal et al., 2019).”
Bootstrap Latent Representations for Multi-modal Recommendation (2022, methodology): “SSL-based methods have achieved competitive results in various CV and NLP tasks [12, 20].”
Self-Supervised Representation Learning for Remote Sensing Image Change Detection Based on Temporal Prediction (2020, background): “In fact, self-supervised learning algorithms, as a branch of unsupervised learning algorithms, have been very early put forward to learn general feature representations from unlabeled image and video data without using any manual labels [26].”
The color out of space: learning self-supervised representations for Earth Observation imagery (2020, methodology): “Unsupervised and self-supervised methods were introduced to learn general visual features from unlabeled data [22].”