A Theoretical Analysis of Contrastive Unsupervised Representation Learning

By Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, Nikunj Saunshi. 2019

910 citations recorded by the source corpus.

Recent empirical works have successfully used unlabeled data to learn feature representations that are broadly useful in downstream classification tasks. Several of these methods are reminiscent of the well-known word2vec embedding algorithm: leveraging availability of pairs of semantically "similar" data points and "negative samples," the learner forces the inner product of representations of similar pairs with each other to be higher on average than with negative samples. The current paper uses the term contrastive learning for such algorithms and presents a theoretical framework for analyzing them by introducing latent classes and hypothesizing that semantically similar points are sampled from the same latent class. This framework allows us to show provable guarantees on the performance of the learned representations on the average classification task that is comprised of a subset of the same set of latent classes. Our generalization bound also shows that learned representations can reduce (labeled) sample complexity on downstream tasks. We conduct controlled experiments in both the text and image domains to support the theory.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “These methods need careful treatment of negative pairs [13] by either relying on large batch sizes [8, 12], memory banks [9] or customized mining strategies [14, 15] to retrieve the negative pairs.”
Contrastive Multiview Coding (2019, background): “As proposed by previous literature (Oord et al., 2018; Hjelm et al., 2019; Arora et al., 2019), the quality of such a representation is evaluated by freezing the weights of encoder and training linear or non-linear classifiers on top of each layer.”
On Mutual Information Maximization for Representation Learning (2019, background): “While it was shown that learning good representations under this protocol can lead to reduced sample complexity for downstream tasks (Arora et al., 2019), some recent works (Bachman et al.”
An Unsupervised Autoregressive Model for Speech Representation Learning (2019, background): “However, there has been evidence [2] and theories [3] supporting the idea that low contrastive loss implies the existence of a linear classifier with low error.”
Prototypical Contrastive Learning of Unsupervised Representations (2020, background): “pairs that share similar semantic meaning and should be closer in the embedding space. However, they are undesirably pushed apart by the contrastive loss. Such problem is deﬁned as class collision in [13] and is shown to hurt representation learning. Essentially, instance discrimination learns an em”
Differentially Private Meta-Learning (2019, methodology): “Datasets: We train a LSTM-RNN for next word prediction on two federated datasets: (1) The Shakespeare dataset as preprocessed by (Caldas et al., 2018), and (2) a dataset constructed from 3 , 000 Wikipedia articles drawn from the Wiki-3029 dataset (Arora et al., 2019), where each article is used as a”
Contrastive Representation Distillation (2019, methodology): “To achieve this, in this paper we leverage the family of contrastive objectives (Gutmann & Hyvärinen, 2010; Oord et al., 2018; Arora et al., 2019; Hjelm et al., 2018).”
Co2L: Contrastive Continual Learning (2021, background): “Indeed, the quality of a representation is typically deﬁned as the joint predictive performance with the best possible (linear) downstream clas-siﬁer (see, e.g ., [3], and references therein), but optimal classiﬁers are only rarely learned under continual setups.”
Pre-Trained Models: Past, Present and Future (2021, background): “Towards the recent development of pre-training objective, Saunshi et al. (2019) conduct a theoretical analysis of contrastive unsupervised representation learning.”
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2020, background): “The estimators are also known as contrastive learning (Arora et al., 2019) that measures the representation similarities between the sampled positive and negative pairs.”
Few-Shot Learning via Learning the Representation, Provably (2020, background): “Further directions include investigating unsupervised representation learning via contrastive methods (Arora et al., 2019) and the effectiveness of predicting multiple auxilary tasks (Doersch and Zisserman, 2017).”
A Mutual Information Maximization Perspective of Language Representation Learning (2019, methodology): “Many of these methods are trained to maximize a particular lower bound called InfoNCE (van den Oord et al., 2019)—also known as contrastive learning (Arora et al., 2019).”
Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere (2020, background): “Analysis based on the assumption of latent classes provides nice theoretical insights (Saunshi et al., 2019), but unfortunately has a rather large gap with empirical practices: the result that representation quality suffers with a large number of negatives is inconsistent with empirical…”
Information Leakage in Embedding Models (2020, background): “ls improve upon state-of-the-art stylometry techniques [59, 65] to infer text authorship with very little training data. Using dual encoder embeddings [39] trained with contrastive learning framework [63] and 10 to 50 labeled sentences per author, we show 1.5–3×fold improvement in classifying hundre”
Dissecting Supervised Constrastive Learning (2021, background): “Through the concept of latent classes (i.e., a construction formalizing the notion of semantic similarity), Arora et al. (2019) prove generalization bounds for downstream supervised classiﬁcation, under the assumption that the supervised task is deﬁned on a subset of the latent classes.”
Representation Learning via Invariant Causal Mechanisms (2020, methodology): “Recently, (Saunshi et al., 2019) proposed a learning theoretic framework to analyze the performance of contrastive objectives.”