Improved Baselines with Momentum Contrastive Learning

By Xinlei Chen, Haoqi Fan, Ross B. Girshick, Kaiming He. 2020

3,990 citations recorded by the source corpus.

Contrastive unsupervised learning has recently shown encouraging progress, e.g., in Momentum Contrast (MoCo) and SimCLR. In this note, we verify the effectiveness of two of SimCLR's design improvements by implementing them in the MoCo framework. With simple modifications to MoCo---namely, using an MLP projection head and more data augmentation---we establish stronger baselines that outperform SimCLR and do not require large training batches. We hope this will make state-of-the-art unsupervised learning research more accessible. Code will be made public.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “SimCLRv2 The SimCLRv2 (Chen et al., 2020d) project released pre-trained and fine-tuned models in various settings.”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, positive): “As an example, “MoCo v2” [8], an extension of a prelim-inary version of this manuscript, achieves 71.1% accuracy with R50 (up from 60.6%), given small changes on the data augmentation and output projection head [7].”
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, background): “Especially in contrastive learning for visual representations [39], remarkable progresses have been made in the past year, for example, SimCLR [24], MoCo [40], and MoCo V2 [41].”
Are we done with ImageNet? (2020, methodology): “We start by assembling a set of 19 models [3, 27, 4, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37] which vary along a number of dimensions, including architecture, training objective and usage of external data (see appendix A).”
Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, methodology): “We run experiments with two popular image-only contrastive visual representation learning methods: SimCLR (Chen et al., 2020a) and MoCo v2 (Chen et al., 2020b).”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, methodology): “Motivated by [29], we also incorporate the memory mechanism from MoCo [20], which designates a memory network (with a moving average of weights for stabilization) whose output will be buffered as negative examples.”
An Empirical Study of Training Self-Supervised Vision Transformers (2021, background): “R50, 800-ep MoCo v2 [11] MoCo v2+ [12] MoCo v3 linear acc.”
Exploring Simple Siamese Representation Learning (2020, methodology): “Simple and effective instantiations of contrastive learning have been developed using Siamese networks [37, 2, 17, 8, 9].”
Vector-quantized Image Modeling with Improved VQGAN (2021, methodology): “Similar to findings reported in iGPT (Chen et al., 2020a), the representations from the middle transformer blocks has better linear-probe accuracy (more study can be found in Appendix).”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “In particular, contrastive representation learning encourages representations to be closer between “positive” pairs and further between “negative” pairs; contrastive predictive coding (CPC, [95]), based on multi-class classification, have been commonly used in state-of-the-art methods [37, 16, 18, 1”
Improving Text-to-Image Synthesis Using Contrastive Learning (2021, background): “In the last two years, various contrastive methods [2, 3, 5, 8, 9, 10, 15, 16, 21, 26, 28, 34, 35, 39] for visual representations have been proposed.”
Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, background): “In unsupervised representation learning [5, 7], adding non-linear layers generally improves performance.”
8-bit Optimizers via Block-wise Quantization (2021, methodology): “, 2009), and Moco v2 contrastive image pretraining and linear finetuning (Chen et al., 2020b)5 on ImageNet-1k (Deng et al.”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “A promising development in self-supervised learning is the contrastive loss, with representative works like CPC (van den Oord et al., 2018), SimCLR (Chen et al., 2020a,b) and MoCo (He et al., 2020; Chen et al., 2020c).”
Improved Baselines with Visual Instruction Tuning (2023, methodology): “Inspired by the improved performance in self-supervised learning by changing from a linear projection to an MLP [9, 10], we find that improving the vision-language connector’s representation power with a two-layer MLP can improve LLaVA’s multimodal capabilities, compared with the original linear…”
Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity (2022, methodology): “• Feature affinities computed on semantic features, self-supervised trained on ImageNet (MoCoV2 [11]).”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, positive): “pretraining objective (Class: Classification, Self : MoCo-v2 [Chen et al. 2020]).”