A Survey on Contrastive Self-supervised Learning

By Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, Fillia Makedon. 2020

1,772 citations recorded by the source corpus.

Self-supervised learning has gained popularity because of its ability to avoid the cost of annotating large-scale datasets. It is capable of adopting self-defined pseudolabels as supervision and use the learned representations for several downstream tasks. Specifically, contrastive learning has recently become a dominant component in self-supervised learning for computer vision, natural language processing (NLP), and other domains. It aims at embedding augmented versions of the same sample close to each other while trying to push away embeddings from different samples. This paper provides an extensive review of self-supervised methods that follow the contrastive approach. The work explains commonly used pretext tasks in a contrastive learning setup, followed by different architectures that have been proposed so far. Next, we present a performance comparison of different methods for multiple downstream tasks such as image classification, object detection, and action recognition. Finally, we conclude with the limitations of the current methods and the need for further techniques and future directions to make meaningful progress.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Continual Barlow Twins: Continual Self-Supervised Learning for Remote Sensing Semantic Segmentation (2022, background): “Recently, referring to CV in general, several innovations have been proposed [53], [54], reviewed in many surveys [24], [55], [56], [57], [58].”
Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning (2021, background): “The general goal of contrastive learning is intuitive: the representation of “similar” samples should be mapped close together in the embedding space, while that of “dissimilar” samples should be further away [25, 30].”
XSimGCL: Towards Extremely Simple Graph Contrastive Learning for Recommendation (2022, background): “Intuitively, encouraging the agreement between related graph augmentations can lead to the learned representations invariant to slight structural perturbations and capture the essential information of the original user-item bipartite graph [1], [19].”
Conditional Contrastive Learning: Removing Undesirable Information in Self-Supervised Representations (2021, methodology): “Self-Supervised Learning Self-supervised learning takes pre-deﬁned tasks from unlabeled samples for pretraining representations [19] and uses the learned representations for downstream tasks, such as visual object detection [6, 18], language understanding and Question Answering [9, 23], and…”
A Survey on Deep Semi-Supervised Learning (2021, background): “Self-supervised learning [46], [47], [48] has gained popularity due to its ability to prevent the expense of annotating large-scale datasets.”
Global and Local Contrastive Self-Supervised Learning for Semantic Segmentation of HR Remote Sensing Images (2021, methodology): “Self-supervised learning (SSL) [34]–[36] provides a new paradigm, as shown in Fig.”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “Jaiswal et al. [31] centered their survey on contrastive learning (CL).”
SCEHR: Supervised Contrastive Learning for Clinical Risk Prediction using Electronic Health Records (2021, methodology): “These methods mainly follow a self-supervised strategy [22], [23], which build augmented data with pseudo-labels to deal with the issue of lacking sufficient supervised information.”
Graph Self-Supervised Learning: A Survey (2021, background): “Some existing surveys mainly review from the perspectives of general SSL [34], SSL for CV [19], or self-supervised contrastive learning [35], while this paper purely focuses on SSL for graph-structured data.”
OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learning (2020, background): “Recently, unsupervised methods based on contrastive learning objectives [8, 9, 22, 23, 33, 35, 38, 46, 49, 62, 69, 71] have shown great results.”
Are Graph Augmentations Necessary?: Simple Graph Contrastive Learning for Recommendation (2021, background): “Recently, a resurgence of contrastive learning (CL) [12, 13, 17] has been witnessed in deep representation learning.”
Online Bag-of-Visual-Words Generation for Unsupervised Representation Learning (2020, background): “Recently, unsupervised methods based on contrastive learning objectives [8, 9, 23, 24, 35, 38, 41, 49, 52, 66, 74, 77] have shown great results.”
Self-Supervised Remote Sensing Feature Learning: Learning Paradigms, Challenges, and Future Works (2022, neutral): “invariant and distinguishable visual features [141], [142].”
Self-Supervised Learning in Remote Sensing: A review (2022, background): “While there exist comprehensive surveys of SSL in computer vision [9, 10, 11, 12], a didactic and up-to-date introduction to the remote sensing community is missing.”
Consistent Representation Learning for Continual Relation Extraction (2022, background): “Contrastive learning (CL) aims to make the representations of similar samples map closer to each other in the embedded space, while that of dissimilar samples should be farther away (Jaiswal et al., 2021).”