Learning to Scale Multilingual Representations for Vision-Language Tasks

By Andrea Burns, Derry Tanti Wijaya, Kate Saenko, Bryan A. Plummer. 2020

36 citations recorded by the source corpus.

Current multilingual vision-language models either require a large number of additional parameters for each supported language, or suffer performance degradation as languages are added. In this paper, we propose a Scalable Multilingual Aligned Language Representation (SMALR) that supports many languages with few model parameters without sacrificing downstream task performance. SMALR learns a fixed size language-agnostic representation for most words in a multilingual vocabulary, keeping language-specific features for just a few. We use a masked cross-language modeling loss to align features with context from other languages. Additionally, we propose a cross-lingual consistency module that ensures predictions made for a query and its machine translation are comparable. The effectiveness of SMALR is demonstrated with ten diverse languages, over twice the number supported in vision-language tasks to date. We evaluate on multilingual image-sentence retrieval and outperform prior work by 3-4% with less than 1/5th the training parameters compared to other word embedding methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

MURAL: Multimodal, Multitask Retrieval Across Languages (2021, methodology): “Burns et al. (2020) exploit this to scale multimodal representations to support more languages with a smaller model than prior studies.”
M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training (2021, methodology): “We compare our work with several related work [15, 31, 11, 32, 1], which are trained on downstream task datasets (MSCOCO and Multi30K) directly without pre-training.”
UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training (2021, background): “SMALR [5] proposes a scalable multilingual model to learn visually aligned word embed-dings, for better balance between multilingual capacity and task performance.”
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021, methodology): “Unlike prior image/video-text work that utilizes RNN (Dong et al., 2019; Chen et al., 2020a; Burns et al., 2020; Kim et al., 2020) and inter-modal contrastive objectives (Sigurdsson et al., 2020; Liu et al., 2019; Huang et al., 2019b; Patrick et al., 2021), we employ Transformers to learn contextual”
M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training (2020, methodology): “In particular, SMALR [27] take advantage of machine translation to augment Multi30K and MSCOCO.”
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval (2021, methodology): “We compare LightningDOT against 3 task-speciﬁc methods: S-LIWE (Wehrmann et al., 2019), MULE (Kim et al., 2020) and SMALR (Burns et al., 2020), which all exploit captions in different languages to learn multi-lingual or language-agnostic word embeddings.”
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2021, background): “In a multilingual setup, all languages share the same embedding space along with the visual data [28, 54, 7].”