mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections

By Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng-da Cao and 5 more. 2022

295 citations recorded by the source corpus.

Large-scale pre-trained foundation models have been an emerging paradigm for building artificial intelligence (AI) systems, which can be quickly adapted to a wide range of downstream tasks. This paper presents mPLUG, a new vision-language foundation model for both cross-modal understanding and generation. Most existing pre-trained models suffer from inefficiency and linguistic signal overwhelmed by long visual sequences in cross-modal alignment. To address both problems, mPLUG introduces an effective and efficient vision-language architecture with novel cross-modal skip-connections.mPLUG is pre-trained end-to-end on large-scale image-text pairs with both discriminative and generative objectives. It achieves state-of-the-art results on a wide range of vision-language downstream tasks, including image captioning, image-text retrieval, visual grounding and visual question answering. mPLUG also demonstrates strong zero-shot transferability on vision-language and video-language tasks. The code and pre-trained models are available at https://github.com/alibaba/AliceMind

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Jianfeng Gao
3. Mohit Bansal
4. Lijuan Wang
5. Zicheng Liu
6. Linjie Li
7. Jianfeng Wang
8. Noah A. Smith

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Zhe Gan
3. Cordelia Schmid
4. Dacheng Tao
5. Jitendra Malik
6. Trevor Darrell
7. Xiaogang Wang
8. Jianfeng Gao

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Zicheng Liu
4. Linjie Li
5. Jianfeng Wang
6. Xiaowei Hu
7. Zhengyuan Yang
8. Ce Liu

Citations in the corpus

• GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[60] Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al.”
• TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering (2023, methodology): “Vision-language models The general pre-trained visionlanguage model are GIT-large [57], VILT-B/32 [28], OFAlarge [58], and mPLUG-large [31].”
• mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality (2023, methodology): “Previous attempts [Li et al., 2022, Xu et al., 2023b] in multi-modal learning have demonstrated that joint learning from uni-modal and multi-modal sources can lead to significant improvements owing to the collaboration between different modalities.”
• ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (2024, methodology): “Subsequently, mPLUG-large [29] is employed as the adju-dicator to evaluate the generated images according to the designated questions.”
• A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multi-Modal (2022, methodology): “Although some general multi-modal pretrained transformer (MPT) models, such as VisualBERT [226], ViLBERT [227], mPLUG [228], UNITER [229], and LXMERT [230], can also be adopted for multi-modal KGR. Due to the variance between the multi-modal KGs and other multi-modal data, directly applying the…”
• Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks (2022, methodology): “Notably, we achieve this without the extensive data augmentations methods typically used by SOTA models (Yu et al., 2022a; He et al., 2022).”