Contrastive Representation Learning: A Framework and Review

By Phúc H. Lê Khắc, Graham F. Healy, Alan F. Smeaton. 2020

957 citations recorded by the source corpus.

Contrastive Learning has recently received interest due to its success in self-supervised representation learning in the computer vision domain. However, the origins of Contrastive Learning date as far back as the 1990s and its development has spanned across many fields and domains including Metric Learning and natural language processing. In this paper, we provide a comprehensive literature review and we propose a general Contrastive Representation Learning framework that simplifies and unifies many different contrastive learning methods. We also provide a taxonomy for each of the components of contrastive learning in order to summarise it and distinguish it from other forms of machine learning. We then discuss the inductive biases which are present in any contrastive learning system and we analyse our framework under different views from various sub-fields of Machine Learning. Examples of how contrastive learning has been applied in computer vision, natural language processing, audio processing, and others, as well as in Reinforcement Learning are also presented. Finally, we discuss the challenges and some of the most promising future research directions ahead.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning (2021, background): “The general goal of contrastive learning is intuitive: the representation of “similar” samples should be mapped close together in the embedding space, while that of “dissimilar” samples should be further away [25, 30].”
Dissecting Supervised Constrastive Learning (2021, background): “We focus on works closely linked to our theoretical results of §3; we refer the reader to (Khosla et al., 2020) (and references therein) for additional background on the supervised contrastive loss and to (Le-Khac et al., 2020) for a general survey on contrastive learning.”
Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE (2021, background): “Fueled by the discovery of efﬁcient algorithms and strong performance [1], extensive research has been devoted to this active topic [20].”
Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text (2021, background): “SimCLR As a paradigm of self-supervised representation learning by comparing different samples, contrastive learning (Wu et al., 2018; Le-Khac et al., 2020; He et al., 2020) aims to make the representations of similar samples be mapped close together, while that of dissimilar samples be further away”
Probing Contextual Language Models for Common Ground with Visual Representations (2020, methodology): “4 This is done via InfoNCE (Oord et al., 2018), a loss function commonly used for retrieval and contrastive learning (Le-Khac et al., 2020).”
Natural language processing applied to mental illness detection: a narrative review (2022, background): “…learning techniques, various learning techniques have emerged and accelerated NLP research, such as adversarial training 181 , contrastive learning 182 , joint learning 183 , reinforcement learning 184 and transfer learning 185 , which can also be utilized for the mental illness detection task.”
Self-Supervised Learning from Automatically Separated Sound Scenes (2021, methodology): “For general-purpose audio, including a variety of environmental sounds beyond speech, the majority of works are based on contrastive learning [14, 15, 16, 17, 18, 19], where a representation is learned by comparing pairs of examples selected by some semantically-correlated notion of similarity [20].”
NewsEmbed: Modeling News through Pre-trained Document Representations (2021, background): “Inspired by contrastive learning initiated in computer vision [36], we believe text representations can be learned by enforcing similar elements to be equal and dissimilar elements to be different.”
SCEHR: Supervised Contrastive Learning for Clinical Risk Prediction using Electronic Health Records (2021, background): “Contrastive learning [18], [22], aiming at learning good representations by bringing similar samples closer and push dissimilar samples away from each other through constructing contrastive loss functions, has shown promising results in image classifications [19], [20], medical image understanding [”
Self-Supervised Pretraining Improves Self-Supervised Pretraining (2021, methodology): “Specifically, we use the MoCo algorithm [7] due to its popularity, available code base, reproducible results without multi-TPU core systems, and similarity to other self-supervised algorithms [30].”