Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere

By Tongzhou Wang, Phillip Isola. 2020

2,520 citations recorded by the source corpus.

Contrastive representation learning has been outstandingly successful in practice. In this work, we identify two key properties related to the contrastive loss: (1) alignment (closeness) of features from positive pairs, and (2) uniformity of the induced distribution of the (normalized) features on the hypersphere. We prove that, asymptotically, the contrastive loss optimizes these properties, and analyze their positive effects on downstream tasks. Empirically, we introduce an optimizable metric to quantify each property. Extensive experiments on standard vision and language datasets confirm the strong agreement between both metrics and downstream task performance. Remarkably, directly optimizing for these two metrics leads to representations with comparable or better performance at downstream tasks than contrastive learning. Project Page: this https URL Code: this https URL , this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, methodology): “In analogy to Wang & Isola (2020), a is called the “alignment score” that measures the similarity of matched pairs and b is called the “uniformity penalty” that measures the similarity of unmatched pairs.”
Improving Text-to-Image Synthesis Using Contrastive Learning (2021, background): “In the last two years, various contrastive methods [2, 3, 5, 8, 9, 10, 15, 16, 21, 26, 28, 34, 35, 39] for visual representations have been proposed.”
Continual Learning by Modeling Intra-Class Variation (2022, background): “Fortunately, recent theoretical studies (e.g., neural collapse (Papyan et al., 2020; Lu & Steinerberger, 2022) and uniformity (Wang & Isola, 2020; Liu et al., 2021c)) discover a regularity of hyperspherical uniformity in the representation space.”
How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2021, background): “Uniformity is an important property for good representations (Wang & Isola, 2020).”
Supervised Contrastive Learning (2020, positive): “Consistent with the findings of [42, 51], our analysis and experiments show that this normalization improves top-1 accuracy.”
Barlow Twins: Self-Supervised Learning via Redundancy Reduction (2021, methodology): “We ﬁrst try to normalize the embeddings along the feature dimension so that they lie on the unit sphere, as it is common practice for losses measuring a cosine similarity (Chen et al., 2020a; Grill et al., 2020; Wang & Isola, 2020).”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, methodology): “The anisotropy problem is naturally connected to uniformity (Wang and Isola, 2020), both highlighting that embeddings should be evenly distributed in the space.”
Self-Supervised Learning: Generative or Contrastive (2020, background): “In [139], Wang et al. conduct interesting theoretical analysis on functions of contrastive loss, and split it into two terms: where the ﬁrst term aims at “alignment” and the second aims at “uniformity” of sample vectors on a sphere given the normalization condition.”
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (2021, background): “Similar to empirical observations and theoretical analyses in recent research [14, 16, 55], a more regularized representation space results in better generalization ability in scenarios with limited labels.”
Self-Supervised Learning for Recommender Systems: A Survey (2022, background): “Wang and Isola [99] identify two key properties related to its NT-Xent version: alignment (closeness) of representations from positive pairs and uniformity of the normalized representations on the hypersphere.”
Understanding the Behaviour of Contrastive Loss (2020, methodology): “In [31], the authors find that the uniformity is a significant property in contrastive learning.”
On Feature Decorrelation in Self-Supervised Learning (2021, methodology): “Complete collapse, where the representations collapse into a constant as in Figure 1a, is the most well-known type of collapse and is addressed differently in existing work with carefully chosen implementation details: To name a few, SimCLR [8] and Uniformity[45] use losses that maximize the distanc”
Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive (2024, methodology): “Contrastive learning is widely used [Wang and Liu, 2021, Wang and Isola, 2020, Saunshi et al., 2019, Oord et al., 2018, Chen et al., 2020, He et al., 2020], often for embedding learning applications.”
The Platonic Representation Hypothesis (2024, positive): “• 49 ResNet-18 models trained with Alignment and Uniformity contrastive loss (Wang & Isola, 2020) on ImageNet-100, Places-365, and 47 realistic and synthetic image datasets from Baradad et al. (2021).”
Towards Domain-Agnostic Contrastive Learning (2020, background): “…e.g. (Chopra et al., 2005; Hadsell et al., 2006; Oord et al., 2018; Hénaff et al., 2019; He et al., 2020; Chen et al., 2020b; Tian et al., 2020; Cai et al., 2020; Wang & Isola, 2020), is perhaps the most popular approach that learns to distinguish semantically similar examples over dissimilar ones.”
SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation (2022, methodology): “In order to understand why SimGRACE can work well, we first introduce the analysis tools from [43].”
Dissecting Supervised Constrastive Learning (2021, methodology): “The unsupervised counterpart of the objective we study in this work is analyzed by Wang & Isola (2020) from a probabilistic perspective.”