Momentum Contrast for Unsupervised Visual Representation Learning
By Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross B. Girshick. Computer Vision and Pattern Recognition, 2019
15,360 citations recorded by the source corpus.
We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning as dictionary look-up, we build a dynamic dictionary with a queue and a moving-averaged encoder. This enables building a large and consistent dictionary on-the-fly that facilitates contrastive unsupervised learning. MoCo provides competitive results under the common linear protocol on ImageNet classification. More importantly, the representations learned by MoCo transfer well to downstream tasks. MoCo can outperform its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large margins. This suggests that the gap between unsupervised and supervised representation learning has been largely closed in many vision tasks.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning
View Paper: https://arxiv.org/abs/1911.05722
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Quoc V. Le
- Sergey Levine
- Percy Liang
- Trevor Darrell
- Dacheng Tao
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Representation & Self-Supervised Learning
- Trevor Darrell
- Ishan Misra
- Andrea Vedaldi
- Andrew Zisserman
- Yoshua Bengio
- Dacheng Tao
- Yann LeCun
- Abhinav Kumar Gupta
Citations in the corpus
- Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “The Contriever model is pre-trained using the MoCo contrastive loss (He et al., 2020), and uses unsupervised data only.”
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “We leave exploration of contrastive pre-training (Chen et al., 2020b; He et al., 2020; Bachman et al., 2019; H ´ enaff et al., 2020) to future work.”
- Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Momentum Contrast (MoCo) We include the MoCo-v1 (He et al., 2020) and the MoCo-v2 (Chen et al., 2020e) checkpoints.”
- A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “Other approaches include shuffling data examples (He et al., 2019), or replacing BN with layer norm (Hénaff et al., 2019).”
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, background): “Especially in contrastive learning for visual representations [39], remarkable progresses have been made in the past year, for example, SimCLR [24], MoCo [40], and MoCo V2 [41].”
- Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “ICT is strongly related to contrastive learning (Wu et al., 2018), which has been widely applied in computer vision (Chen et al., 2020; He et al., 2020).”
- Learning Dense Representations of Phrases at Scale (2020, methodology): “This approach is inspired by the momentum contrast idea recently proposed in unsupervised visual representation learning (He et al., 2020).”
- Big Transfer (BiT): General Visual Representation Learning (2019, background): “For example, [13] show that unsupervised representations trained on 1B unlabelled Instagram images transfer comparably or better than supervised ILSVRC-2012 features.”
- Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, methodology): “First of all, similar to previous work on image-only contrastive learning (Chen et al., 2020a; He et al., 2020), the pretraining results are most sensitive to the choice of the temperature value τ .”
- Learning Visual Representations with Caption Annotations (2020, background): “they require either extremely large training sets (up to a hundred million images) [7,24] or need to be trained much longer with larger networks to express their full potential [9,26].”
- Self-Supervised MultiModal Versatile Networks (2020, methodology): “Recently, methods that maximise the similarity between multiple views (augmented versions) of the same image via contrastive losses [12, 16, 32, 33, 34, 60] stand out due to impressive results on the ImageNet benchmark; we draw inspiration from them (e.”
- Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, methodology): “Unsupervised or self-supervised pretraining followed by supervised fine-tuning on a few labeled examples has been extensively used in natural language processing [6, 5, 7–9], but has only shown promising results in computer vision very recently [19, 20, 1].”
- Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “Similarly, we evaluate on object detection by reproducing the setup in [9] using a Faster R-CNN architecture [82], as detailed in Appendix E.”
- Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, methodology): “supervised learning to improve the generalization of the meta-learning algorithms [57] removes the needs miniImageNet 5-way model backbone 1-shot 5-shot Supervised ResNet50 57.56 0.79 73.81 0.63 Moco [16] ResNet50 54.19 0.93 73.04 0.61 CMC [49] ResNet50 56.10 0.89 73.87 0.65 Table 3. Comparsions of ”
- Improved Baselines with Momentum Contrastive Learning (2020, methodology): “The contrastive loss (1) can be minimized by various mechanisms that differ in how the keys are maintained [6].”
- Contrastive Multiview Coding (2019, neutral): “On top of it, strengthening data augmentation with RandAugment [13] yields better or comparable results to other state-of-the-art methods [40,79,87,26,48,18,28,4].”
- Benchmarking Detection Transfer Learning with Vision Transformers (2021, background): “[17]).”