MURAL: Multimodal, Multitask Retrieval Across Languages

By Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting-Li Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, Jason Baldridge. 2021

64 citations recorded by the source corpus.

Both image-caption pairs and translation pairs provide the means to learn deep representations of and connections between languages. We use both types of pairs in MURAL (MUltimodal, MUltitask Representations Across Languages), a dual encoder that solves two tasks: 1) image-text matching and 2) translation pair matching. By incorporating billions of translation pairs, MURAL extends ALIGN (Jia et al. PMLR'21)--a state-of-the-art dual encoder learned from 1.8 billion noisy image-text pairs. When using the same encoders, MURAL's performance matches or exceeds ALIGN's cross-modal retrieval performance on well-resourced languages across several datasets. More importantly, it considerably improves performance on under-resourced languages, showing that text-text learning can overcome a paucity of image-caption examples for these languages. On the Wikipedia Image-Text dataset, for example, MURAL-base improves zero-shot mean recall by 8.1% on average for eight under-resourced languages and by 6.8% on average when fine-tuning. We additionally show that MURAL's text representations cluster not only with respect to genealogical connections but also based on areal linguistics, such as the Balkan Sprachbund.

Subfields: Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

SLIP: Self-supervision meets Language-Image Pre-training (2021, background): “MURAL [30] extends ALIGN to the multi-lingual setting and introduces a cross-lingual objective to improve multi-lingual image and text retrieval.”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, methodology): “, CLIP [65]) are trained on joint image(s) and corresponding text captions via variants of masked language modeling (MLM) objective [66, 67, 68, 69], contrastive loss [70, 71, 65] or other supervised objectives[72, 73].”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “For example, in terms of linguistic data, visual-language models (VLMs) (Wang et al. 2021; Jain et al. 2021) are trained on image and video captions, but large language models (LMs) (Devlin et al. 2018; Thoppilan et al. 2022; Chen et al. 2021) are additionally trained on a large corpora of other…”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[49] Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, and Jason Baldridge.”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “34 MURAL [155] arXiv-2021 image-text Trans CS Adopt multi-task contrastive learning objective (image-text, text-text) 430M -”
Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, background): “There are an increasing number of studies to improve their generality from various modeling perspectives, including training objectives [11,13,14,38,65,68], scaling techniques [9,40,65], data efficiency [25, 28], and leveraging multilingual correlations [9, 19].”