M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training

By Haoyang Huang, Lin Su, Di Qi, Nan Duan, Edward Cui, Taroon Bharti, Lei Zhang, Lijuan Wang, Jianfeng Gao, Bei Liu and 4 more. 2021

120 citations recorded by the source corpus.

We present M3P, a Multitask Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training into a unified framework via multitask pre-training. Our goal is to learn universal representations that can map objects occurred in different modalities or texts expressed in different languages into a common semantic space. In addition, to explicitly encourage fine-grained alignment between images and non-English languages, we also propose Multimodal Code-switched Training (MCT) to combine monolingual pre-training and multimodal pre-training via a code-switch strategy. Experiments are performed on the multilingual image retrieval task across two benchmark datasets, including MSCOCO and Multi30K. M3P can achieve comparable results for English and new state-of-the-art results for non-English languages.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

MURAL: Multimodal, Multitask Retrieval Across Languages (2021, methodology): “More recent work learns cross encoder models with multitask training objectives (Ni et al., 2021; Zhou et al., 2021); in these, a single multimodal encoder attends to both inputs and exploits deep associations between images and captions.”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, methodology): “We evaluate the zero-shot model performance of ALIGN and compare it with M 3 P (Huang et al., 2020a) and UC2 (Zhou et al., 2021).”
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages (2022, methodology): “1 The need to expand V&L research towards more languages has been recognised by 1) the recent creation of multilingual training and evaluation data across diverse V&L tasks and languages (Armitage et al., 2020; Srinivasan et al., 2021; Su et al., 2021; Pfeiffer et al., 2022; Liu et al., 2021; Wang e”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, unknown): “62 M3P [175] CVPR-2021 image-text BERT xMLM, MC-MLM, MC-MRM Multitask, Multilingual, Multimodal Pre-training URL”
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval (2021, background): “…et al., 2019; Liu et al., 2019; Yang et al., 2019; Raffel et al., 2020; Lan et al., 2020; Clark et al., 2020), vision-and-language pre-training (Huang et al., 2020b; Su et al., 2020; Li et al., 2020b, 2019a) has become the prevailing paradigm in learning multimodal representations, with strong…”
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2021, methodology): “In all the experiments, we opt for state-of-the-art pretrained multi-modal models for monolingual (English) and multilingual contexts: OSCAR [32] and M3P [24], respectively.”
Multimodal Learning With Transformers: A Survey (2022, background): “, universal cross-lingual generalization from English to non-English multimodal contexts [206], [277].”
Language-Agnostic Visual Embeddings for Cross-Script Handwriting Retrieval (2026, unknown): “Pioneering works like CLIP [Radford et al. , 2021], ALIGN [Jia et al. , 2021], and their multilingual variants (e.g., M 3 P [Ni et al. , 2021], UC 2 [Zhou et al. , 2021], MURAL [Jain et al. , 2021]) established the vision-language alignment paradigm via contrastive learning, extending to…”