UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training

By Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu, Jingjing Liu. 2021

112 citations recorded by the source corpus.

Vision-and-language pre-training has achieved impressive success in learning multimodal representations between vision and language. To generalize this success to non-English languages, we introduce UC2, the first machine translation-augmented framework for cross-lingual cross-modal representation learning. To tackle the scarcity problem of multilingual captions for image datasets, we first augment existing English-only datasets with other languages via machine translation (MT). Then we extend the standard Masked Language Modeling and Image-Text Matching training objectives to multilingual setting, where alignment between different languages is captured through shared visual context (i.e., using image as pivot). To facilitate the learning of a joint embedding space of images and all languages of interest, we further propose two novel pre-training tasks, namely Masked Region-to-Token Modeling (MRTM) and Visual Translation Language Modeling (VTLM), leveraging MT-enhanced translated data. Evaluation on multilingual image-text retrieval and multilingual visual question answering benchmarks demonstrates that our proposed framework achieves new state of the art on diverse non-English benchmarks while maintaining comparable performance to monolingual pre-trained models on English tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Pascale Fung
2. Mohit Bansal
3. Jason Baldridge
4. Jianfeng Gao
5. Yinfei Yang
6. Zhou Yu
7. Graham Neubig
8. Quoc V. Le

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Trevor Darrell
4. Ming-Hsuan Yang
5. Cordelia Schmid
6. Andrew Zisserman
7. Chen Change Loy
8. Xiaogang Wang

In Multimodal Learning

1. Jason Baldridge
2. Zhou Yu
3. Yinfei Yang
4. Yuhao Cui
5. Wenliang Dai
6. Jun Yu
7. Pascale Fung
8. Dan Su

In Computational Linguistics & Semantics

1. Jason Baldridge
2. Hieu Pham
3. Quoc V. Le

Citations in the corpus

• MURAL: Multimodal, Multitask Retrieval Across Languages (2021, methodology): “More recent work learns cross encoder models with multitask training objectives (Ni et al., 2021; Zhou et al., 2021); in these, a single multimodal encoder attends to both inputs and exploits deep associations between images and captions.”
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, methodology): “We evaluate the zero-shot model performance of ALIGN and compare it with M 3 P (Huang et al., 2020a) and UC2 (Zhou et al., 2021).”
• ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration (2021, background): “A concurrent work UC2 utilizes off-the-shelf machine translation model to construct aligned multilingual dataset for texts and regard this extra information as prior knowledge to enhance the learning of cross-modal semantic alignment [55].”
• Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training (2022, methodology): “ITM is a widely used objective in VLP (Li et al., 2020a; Chen et al., 2020; Zhou et al., 2021).”
• Language-Agnostic Visual Embeddings for Cross-Script Handwriting Retrieval (2026, unknown): “Pioneering works like CLIP [Radford et al. , 2021], ALIGN [Jia et al. , 2021], and their multilingual variants (e.g., M 3 P [Ni et al. , 2021], UC 2 [Zhou et al. , 2021], MURAL [Jain et al. , 2021]) established the vision-language alignment paradigm via contrastive learning, extending to…”