Prototypical Contrastive Learning of Unsupervised Representations

By Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, Steven C. H. Hoi. 2020

1,199 citations recorded by the source corpus.

This paper presents Prototypical Contrastive Learning (PCL), an unsupervised representation learning method that addresses the fundamental limitations of instance-wise contrastive learning. PCL not only learns low-level features for the task of instance discrimination, but more importantly, it implicitly encodes semantic structures of the data into the learned embedding space. Specifically, we introduce prototypes as latent variables to help find the maximum-likelihood estimation of the network parameters in an Expectation-Maximization framework. We iteratively perform E-step as finding the distribution of prototypes via clustering and M-step as optimizing the network via contrastive learning. We propose ProtoNCE loss, a generalized version of the InfoNCE loss for contrastive learning, which encourages representations to be closer to their assigned prototypes. PCL achieves state-of-the-art results on multiple unsupervised representation learning benchmarks, with >10% accuracy improvement in low-resource transfer tasks. Code is available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, background): “Our work builds upon recent success on contrastive learning of visual representations [44, 16, 17, 45, 18, 19, 46, 42, 20, 47, 48, 1], a sub-area within self-supervised learning.”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “Among discriminative methods, contrastive methods [9, 10, 32, 33, 34, 11, 35, 36] currently achieve state-of-the-art performance in self-supervised learning [37, 8, 38, 12].”
VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “– Self-supervised pretraining: We choose three recent methods based on their availability and compatibility with our evaluation setup – MoCo [24], PCL [57], and SwAV [58].”
BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “Another strand of methods use clustering to organize image examples (Caron et al., 2018; Asano et al., 2020; Caron et al., 2020; Li et al., 2021).”
Emerging Properties in Self-Supervised Vision Transformers (2021, background): “Several variants allow automatic grouping of instances in the form of clustering [2, 8, 9, 36, 42, 74, 80, 85].”
Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “These methods discriminate either between each instance feature [8, 22, 37] or between their cluster assignments [2, 7, 32].”
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, methodology): “The recent CLIP [6] and ALIGN [7] perform pre-training on massive noisy web data using a contrastive loss, one of the most effective loss for representation learning [24, 25, 26, 27].”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, background): “Recently, self-supervised (Hénaff et al., 2019; Chen et al., 2020b; He et al., 2020; Misra & Maaten, 2020; Li et al., 2021; Grill et al., 2020; Caron et al., 2020) and semi-supervised learning (Yalniz et al.”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, positive): “We mostly follow hyperparameters from PCL [36]: we train during 20 epochs with a batch size of 256 , we use distinct learning rates for the convnet weights and the ﬁnal linear layer, and we decay the learning rates by a factor 0 .”
Class Gradient Projection For Continual Learning (2022, methodology): “Prototypical Contrastive Learning (PCL) [23] uses the centroids of clusters as prototypes, and pulls the image embedding closer to its prototypes.”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, background): “There also have been works that utilize the contrastive loss [12, 30, 48]; especially, SimCLR [12] proposes a simplified contrastive learning method, which enables representation learning by pulling the randomly transformed samples from the same image closer while pushing ones apart from other image”
Self-Supervised Training Enhances Online Continual Learning (2021, background): “More recent works use contrastive learning [26, 51], where the model learns which datapoints are similar or different based on feature similarity, and they have even surpassed supervised networks on many downstream tasks [30, 14, 16, 15, 38, 25, 9].”
Meta Pseudo Labels (2020, background): “Self-supervised losses typically encourage the model to develop a common sense about images, such as in-painting [50], solving jigsaw puzzles [47], predicting the rotation angle [19], contrastive prediction [25, 10, 8, 9, 38], or bootstraping the latent space [21].”
Support-set bottlenecks for video-text representation learning (2020, background): “…in the NLP domain, several works have leveraged transformers to learn transferable image (Chen et al., 2020a; Desai & Johnson, 2020; Sariyildiz et al., 2020) or multi-modal image-text representations (Chen et al., 2019; Li et al., 2020; 2019; Lu et al., 2019; Su et al., 2019; Tan & Bansal, 2019).”
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (2021, background): “Li et al. (2021) 54.5 - BYOL Grill et al. (2020) 66.7 64.9 SwAV Caron et al. (2020) 65.7 62.7 Barlow Twins Zbontar et al. (2021) 64.8 62.9 VICReg 64.5 62.8 VICReg borrows the decorrelation mechanism of Barlow Twins Zbontar et al. (2021) and we observe that it therefore has the same dependency on…”
Intent Contrastive Learning for Sequential Recommendation (2022, methodology): “Contrastive Self-Supervised Learning (SSL) has brought much attentions by different research communities including CV [2, 4, 9, 14, 20] and NLP [7, 8, 29, 50], as well as recommendation [44, 45, 47, 51].”