Multimodal Contrastive Training for Visual Representation Learning

By Xin Yuan, Zhe L. Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, Baldo Faieta. 2021

217 citations recorded by the source corpus.

We develop an approach to learning visual representations that embraces multimodal data, driven by a combination of intra- and inter-modal similarity preservation objectives. Unlike existing visual pre-training methods, which solve a proxy prediction task in a single domain, our method exploits intrinsic data properties within each modality and semantic information from cross-modal correlation simultaneously, hence improving the quality of learned visual representations. By including multimodal training in a unified framework with different types of contrastive losses, our method can learn more powerful and generic visual features. We first train our model on COCO and evaluate the learned visual representations on various downstream tasks including image classification, object detection, and instance segmentation. For example, the visual representations pre-trained on COCO by our method achieve state-of-the-art top-1 validation accuracy of 55.3% on ImageNet classification, under the common transfer protocol. We also evaluate our method on the large-scale Stock images dataset and show its effectiveness on multi-label image tagging, and cross-modal retrieval tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chen Change Loy
2. Phillip Isola
3. Yonglong Tian
4. Dilip Krishnan
5. Alexander Kirillov
6. Saining Xie
7. Bo Dai
8. Jiashi Feng

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Chen Change Loy
3. Ming-Hsuan Yang
4. Dacheng Tao
5. Xiaogang Wang
6. Trevor Darrell
7. Jitendra Malik
8. Chunhua Shen

In Representation & Self-Supervised Learning

1. Phillip Isola
2. Yonglong Tian
3. Chen Change Loy
4. Dilip Krishnan
5. Saining Xie
6. Alexander Kirillov
7. Norman Mu
8. David A. Wagner

Citations in the corpus

• SLIP: Self-supervision meets Language-Image Pre-training (2021, background): “Prior to CLIP, Multimodal Contrastive Training [62] adds contrastive image-image and language-image losses to VirTex which further improve performance.”
• Extract Free Dense Labels from CLIP (2021, background): “Some studies [44,16,17,48,12,53] propose to learn from image-caption pairs.”
• Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling (2021, background): “Recent works [76, 77] improve training efficiency by removing the need for object detectors and scale to hundreds of millions of samples for substantial performance gains [78].”
• DenseCLIP: Extract Free Dense Labels from CLIP (2021, background): “Some studies [12, 16, 17, 44, 47, 53] pro-pose to learn from image-caption pairs.”
• Improving CLIP Training with Language Rewrites (2023, background): “There are a number of earlier works demonstrating the effectiveness of learning visual representations from the supervision of corresponding text [25, 31, 14, 62].”
• Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “RATP [174] uses the Monte-Carlo Tree Search (MCTS) to perform multiple simulations, identifying optimal retrieval content.”