Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models

By Po-Yao (Bernie) Huang, Mandela Patrick, Junjie Hu, Graham Neubig, Florian Metze, Alexander G. Hauptmann. 2021

64 citations recorded by the source corpus.

This paper studies zero-shot cross-lingual transfer of vision-language models. Specifically, we focus on multilingual text-to-video search and propose a Transformer-based model that learns contextual multilingual multimodal embeddings. Under a zero-shot setting, we empirically demonstrate that performance degrades significantly when we query the multilingual text-video model with non-English sentences. To address this problem, we introduce a multilingual multimodal pre-training strategy, and collect a new multilingual instructional video dataset (Multi-HowTo100M) for pre-training. Experiments on VTT show that our method significantly improves video search in non-English languages without additional annotations. Furthermore, when multilingual annotations are available, our method outperforms recent baselines by a large margin in multilingual text-to-video search on VTT and VATEX; as well as in multilingual text-to-image search on Multi30K. Our model and Multi-HowTo100M is available at http://github.com/berniebear/Multi-HT100M.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Zhe Gan
3. Jianfeng Gao
4. Linjie Li
5. Jingjing Liu
6. Lijuan Wang
7. Zicheng Liu
8. Yu Cheng

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Zhe Gan
3. Cordelia Schmid
4. Mohit Bansal
5. Dacheng Tao
6. Trevor Darrell
7. Andrew Zisserman
8. Ming-Hsuan Yang

In Multimodal Learning

1. Zhe Gan
2. Linjie Li
3. Lijuan Wang
4. Zicheng Liu
5. Jingjing Liu
6. Mohit Bansal
7. Yu Cheng
8. Jianfeng Gao

Citations in the corpus

• VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021, neutral): “For example, (Miech et al., 2019, 2020; Rouditchenko et al., 2020; Ging et al., 2020; Gabeur et al., 2020; Alayrac et al., 2020; Patrick et al., 2021; Huang et al., 2021) adopts a retrieval task for pre-training.”
• Support-set bottlenecks for video-text representation learning (2020, neutral): “, 2019; Tan & Bansal, 2019) and video-multilingual text (Huang et al., 2021) representations.”
• VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “Future work could consider multilingual VidL datasets [63, 54, 24, 30] as a complementary evaluation to further test systems’ ability on processing information in different languages.”
• VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, background): “For example, (Miech et al., 2019, 2020; Ging et al., 2020; Gabeur et al., 2020; Alayrac et al., 2020; Patrick et al., 2021; Huang et al., 2021) adopt contrastive learning for pre-training and shows the possibility of zero-shot transfer to text-video retrieval tasks.”
• Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers (2021, background): “Transformers have also been used in a variety of vision image tasks, such as image representation learning [12, 82, 18, 59], image generation [51], object detection [48, 10], video question-answering [35], few-shot learning [19], and image–text [49, 63, 67, 44, 68], video-text [65, 64, 89, 25, 53, 2”
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages (2022, background): “, 2021), multilingual text-to-video search (Huang et al., 2021; Lei et al., 2021), multilingual visual reasoning (Liu et al.”