M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training

By Minheng Ni, Haoyang Huang, Lin Su, Edward Cui, Taroon Bharti, Lijuan Wang, Jianfeng Gao, Dongdong Zhang, Nan Duan. 2020

7 citations recorded by the source corpus.

This paper presents a Multitask Multilingual Multimodal Pre-trained model (M3P) that combines multilingual-monomodal pre-training and monolingual-multimodal pre-training into a unified framework via multitask learning and weight sharing. The model learns universal representations that can map objects that occurred in different modalities or expressed in different languages to vectors in a common semantic space. To verify the generalization capability of M3P, we fine-tune the pre-trained model for different types of downstream tasks: multilingual image-text retrieval, multilingual image captioning, multimodal machine translation, multilingual natural language inference and multilingual text generation. Evaluation shows that M3P can (i) achieve comparable results on multilingual tasks and English multimodal tasks, compared to the state-of-the-art models pre-trained for these two types of tasks separately, and (ii) obtain new state-of-the-art results on non-English multimodal tasks in the zero-shot or few-shot setting. We also build a new Multilingual Image-Language Dataset (MILD) by collecting large amounts of (text-query, image, context) triplets in 8 languages from the logs of a commercial search engine

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Minlie Huang
3. Xipeng Qiu
4. Mohit Bansal
5. Jianfeng Gao
6. Jingjing Liu
7. Maosong Sun
8. Linjie Li

In Computer Vision and Pattern Recognition

1. Linjie Li
2. Jingjing Liu
3. Yu Cheng
4. Zhe Gan
5. Jianfeng Gao
6. Cordelia Schmid
7. Luc Van Gool
8. Dacheng Tao

Citations in the corpus

• Pre-Trained Models: Past, Present and Future (2021, background): “Additionally, some researchers found that they could get even better performance on benchmarks when training one model with several languages comparing with training several monolingual models (Lample and Conneau, 2019; Huang et al., 2020b).”
• UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training (2021, methodology): “A pioneering work is M 3 P [24], which formulates the training process as alternating V+L pre-training between cross-modal monolingual corpus and mono-modal cross-lingual corpus.”