What makes for good views for contrastive learning

By Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, Phillip Isola. 2020

1,584 citations recorded by the source corpus.

Contrastive learning between multiple views of the data has recently achieved state of the art performance in the field of self-supervised representation learning. Despite its success, the influence of different view choices has been less studied. In this paper, we use empirical analysis to better understand the importance of view selection, and argue that we should reduce the mutual information (MI) between views while keeping task-relevant information intact. To verify this hypothesis, we devise unsupervised and semi-supervised frameworks that learn effective views by aiming to reduce their MI. We also consider data augmentation as a way to reduce MI, and show that increasing data augmentation indeed leads to decreasing MI and improves downstream classification accuracy. As a by-product, we also achieve a new state-of-the-art accuracy on unsupervised pre-training for ImageNet classification ($73\%$ top-1 linear readoff with a ResNet-50). In addition, transferring our models to PASCAL VOC object detection and COCO instance segmentation consistently outperforms supervised pre-training. Code:this http URL

Subfields: Education, Sociology and Political Science, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Education

In Sociology and Political Science

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Supervised MultiModal Versatile Networks (2020, background): “However, details of view generation are crucial and require careful design [81].”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “In addition, their performance critically depends on the choice of image augmentations [34, 11, 8, 12].”
Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, methodology): “Using unsupervised learning [56, 46, 17, 48] to improve the generalization of the meta-learning algorithms [53] removes the needs of data annotation.”
Contrastive Multiview Coding (2019, background): “Selecting optimal views, which just share relevant signal, has been further discussed in a follow-up work [75] of CMC, and may be a fruitful direction for future research.”
Improving Text-to-Image Synthesis Using Contrastive Learning (2021, background): “In the last two years, various contrastive methods [2, 3, 5, 8, 9, 10, 15, 16, 21, 26, 28, 34, 35, 39] for visual representations have been proposed.”
Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, background): “Given two random variables v1 and v2, often known as views of the data, contrastive learning aims to find useful representations of v1 and v2 by learning a function that measures the dependence of two views [53], i.”
Combined Scaling for Zero-shot Transfer Learning (2021, background): “Lastly, scaling up batch sizes is also the key for improving the model effectiveness (Goyal et al., 2017), especially for the contrastive loss (Chen et al., 2020a; Tian et al., 2020; Jia et al., 2021; Radford et al., 2021).”
Unified Contrastive Learning in Image-Text-Label Space (2022, background): “Contrastive learning has laid the foundation for the best performing SSL models [3, 6, 8, 21, 24, 57, 58].”
Supervised Contrastive Learning (2020, methodology): “Table 3: Top-1/Top-5 accuracy results on ImageNet for AutoAugment [5] with ResNet-50 and for Stacked RandAugment [49] with ResNet-101 and ResNet-200.”
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (2021, background): “…over the last years, almost reaching the performance of supervised baselines on many downstream tasks Bachman et al. (2019); Misra & Maaten (2020); He et al. (2020); Tian et al. (2020); Caron et al. (2020); Grill et al. (2020); Chen & He (2020); Gidaris et al. (2021); Zbontar et al. (2021).”
How Well Do Self-Supervised Models Transfer? (2020, background): “Contrastive: InsDis (also known as NPID) [58], MoCo-v1 [21] and its upgrade MoCo-v2 [8], PIRL [39], SimCLRv1 [6] and SimCLR-v2 [7], InfoMin [52] and BYOL [17].”
Spatiotemporal Contrastive Video Representation Learning (2020, background): “Tian et al . study what makes a good view in this contrastive learning framework [38].”
Complementary Relation Contrastive Distillation (2021, background): “Contrastive Learning serves as the core idea of several recent works on self-supervised representation learning [8, 16, 30, 20, 15, 44, 42, 13].”
Self-Supervised Learning for Recommender Systems: A Survey (2022, background): “Prior studies [55], [56], [57] have highlighted the crucial role played by data augmentation in facilitating the learning of high-quality, generalizable representations.”