WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training

By Yuqi Huo, Manli Zhang, Guangzhen Liu, Haoyu Lu, Yizhao Gao, Guoxing Yang, Jing Wen, Baogui Xu, Weihao Zheng, Zongzheng Xi and 22 more. 2021

146 citations recorded by the source corpus.

Multi-modal pre-training models have been intensively explored to bridge vision and language in recent years. However, most of them explicitly model the cross-modal interaction between image-text pairs, by assuming that there exists strong semantic correlation between the text and image modalities. Since this strong assumption is often invalid in real-world scenarios, we choose to implicitly model the cross-modal correlation for large-scale multi-modal pre-training, which is the focus of the Chinese project `WenLan' led by our team. Specifically, with the weak correlation assumption over image-text pairs, we propose a two-tower pre-training model called BriVL within the cross-modal contrastive learning framework. Unlike OpenAI CLIP that adopts a simple contrastive learning method, we devise a more advanced algorithm by adapting the latest method MoCo into the cross-modal scenario. By building a large queue-based dictionary, our BriVL can incorporate more negative samples in limited GPU resources. We further construct a large Chinese multi-source image-text dataset called RUC-CAS-WenLan for pre-training our BriVL model. Extensive experiments demonstrate that the pre-trained BriVL model outperforms both UNITER and OpenAI CLIP on various downstream tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Zhe Gan
3. Jianfeng Gao
4. Minlie Huang
5. Lijuan Wang
6. Zicheng Liu
7. Jianfeng Wang
8. Mohit Bansal

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Yi Yang
3. Dacheng Tao
4. Xiaogang Wang
5. Ming-Hsuan Yang
6. Wanli Ouyang
7. Cordelia Schmid
8. Trevor Darrell

In Multimodal Learning

1. Lijuan Wang
2. Jianfeng Wang
3. Zicheng Liu
4. Zhe Gan
5. Zhengyuan Yang
6. Xiaowei Hu
7. Yumao Lu
8. Xiaodan Liang

Citations in the corpus

• Scaling Up Vision-Language Pretraining for Image Captioning (2021, methodology): “Besides CC12M, there also exist some other largescale image-text datasets, such as WIT [36], WenLan [16], LAION-400M [34], and the datasets used in CLIP [33] and ALIGN [17].”
• UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning (2020, neutral): “WenLan (Huo et al., 2021) further proposes a similar two-tower Chinese multi-modal pre-training model and adapts MoCo (He et al., 2020) to improve the contrastive cross-modal learning process.”
• Pre-Trained Models: Past, Present and Future (2021, methodology): “The success of this kind of holistic alignment can be largely attributed to the enlarging scale of web data, which is 400 million image-text pairs for CLIP and 30 million for WenLan.”
• Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm (2021, background): “Moreover, this series of models (Radford et al., 2021; Jia et al., 2021; Huo et al., 2021) can perform zero-shot recognition, adapting to new categories with no seen labeled data.”
• Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, unknown): “Huo et al. [62] propose WenLan to address this issue by using two-tower architecture and cross-modal contrastive learning based on MoCo [63], which can leverage more negative samples in limited GPU resources.”
• Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “Representative largescale MM-PTMs based on cross-stream structure include BriVL [106] and CLIP [77], etc.”
• Prompt-aligned Gradient for Prompt Tuning (2022, methodology): “The pre-text task for pre-training Vision-Language Models (VLMs) mainly falls into 4 categories: 1) Masked language modeling [18, 25], 2) Masked region prediction [47, 46], 3) Image-text matching [47, 18], and 4) Contrastive learning [36, 16, 21, 15].”
• CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation (2023, methodology): “Some single-modal [22] or vison-language [26] pretraining works also adopt momentum model as a temporal ensembling method [19] to smoothly guide the training.”
• Multimodal Learning With Transformers: A Survey (2022, unknown): “Various recently released datasets are million-scale, e.g., Product1M [137], Conceptual 12M [138], RUC-CAS-WenLan [139] (30 M), HowToVQA69M [140], HowTo100M [141], ALT200M [142], LAION-400M [143].”
• X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval (2022, methodology): “To better cope with the image-text retrieval tasks, contrastive language-image pre-training methods such as CLIP [44], ALIGN [23] and WenLan [19] have been proposed, by leveraging billion-scale image-text pairs data from the web with a dual-stream Transformer.”