Vision-Language Pre-Training with Triple Contrastive Learning

By Jinyu Yang, Jiali Duan, Son Dinh Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul M. Chilimbi, Junzhou Huang. Computer Vision and Pattern Recognition, 2022

392 citations recorded by the source corpus.

Vision-language representation learning largely benefits from image-text alignment through contrastive losses (e.g., InfoNCE loss). The success of this alignment strategy is attributed to its capability in maximizing the mutual information (MI) between an image and its matched text. However, simply performing cross-modal alignment (CMA) ignores data potential within each modality, which may result in degraded representations. For instance, although CMA-based models are able to map image-text pairs close together in the embedding space, they fail to ensure that similar inputs from the same modality stay close by. This problem can get even worse when the pre-training data is noisy. In this paper, we propose triple contrastive learning (TCL) for vision-language pre-training by leveraging both cross-modal and intra-modal self-supervision. Besides CMA, TCL introduces an intra-modal contrastive objective to provide complementary benefits in representation learning. To take advantage of localized and structural information from image and text input, TCL further maximizes the average MI between local regions of image/text and their global summary. To the best of our knowledge, ours is the first work that takes into account local structure information for multi-modality representation learning. Experimental evaluations show that our approach is competitive and achieves the new state of the art on various common downstream vision-language tasks such as image-text retrieval and visual question answering.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0ad84c4bf7499df6945fc51b24ae2ac779f218ec

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chunyuan Li
2. Yong Jae Lee
3. Jianfeng Gao
4. Jianwei Yang
5. Haotian Liu
6. Ce Liu
7. Zhe Gan
8. Kilho Son

In Computer Vision and Pattern Recognition

1. Yong Jae Lee
2. Jianfeng Gao
3. Luc Van Gool
4. Chunyuan Li
5. Yi Yang
6. Dacheng Tao
7. Jianwei Yang
8. Trevor Darrell

Citations in the corpus

• Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “65 TCL [178] CVPR-2022 image-text BERT CMA, IMC, LMI ITM, MLM The first work considers local structure information for multi-modality representation learning 123.”
• Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, background): “CC3M [47], CC12M [4], YFCC15M [43, 51]), by exploring pretraining on a unified image-text-label space [59], token-level contrastive loss [61], and auxiliary within-modality contrastive loss [29,37,58,63].”
• MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, neutral): “…pushing the performance boundaries across a spectrum of downstream tasks (Li et al., 2020; Akbari et al., 2021; Fang et al., 2021; Yan et al., 2021; Li et al., 2021; Radford et al., 2021; Li et al., 2022; Zellers et al., 2022; Zeng et al., 2022b; Yang et al., 2022; Wang et al., 2022a,b).”
• Multimodal Learning With Transformers: A Survey (2022, neutral): “, video-text [7], [107], [117], image-text [102], [103], [104], [193], [194], [195], acoustictext [180].”