Image Pivoting for Learning Multilingual Multimodal Representations

By Spandana Gella, Rico Sennrich, Frank Keller, Mirella Lapata. 2017

79 citations recorded by the source corpus.

In this paper we propose a model to learn multimodal multilingual representations for matching images and sentences in different languages, with the aim of advancing multilingual versions of image search and image understanding. Our model learns a common representation for images and their descriptions in two different languages (which need not be parallel) by considering the image as a pivot between two languages. We introduce a new pairwise ranking loss function which can handle both symmetric and asymmetric similarity between the two modalities. We evaluate our models on image-description ranking for German and English, and on semantic textual similarity of image descriptions in English. In both cases we achieve state-of-the-art performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

MURAL: Multimodal, Multitask Retrieval Across Languages (2021, background): “Multilingual captions for images provide indirect but valuable associations between languages (Gella et al., 2017).”
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021, background): “Finally, while joint-space approaches to multimodal training are applied to multilingual data (Gella et al., 2017; Sigurdsson et al., 2020), all existing multimodal transformers are applied to English; an interesting future direction is to extend these models to other languages.”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “Gouws and Søgaard (2015) create a pseudo-crosslingual corpus by replacing words based on part-of-speech equivalence, that is, words with the same part-of-speech in different languages are replaced with one another.”
Zero-Resource Neural Machine Translation with Multi-Agent Communication Game (2018, background): “Gella et al. (2017) propose to learn multimodal multilingual representations of fixed length for matching images and sentences in different languages in the same space with image as a pivot.”
COCO-CN for Cross-Lingual Image Tagging, Captioning, and Retrieval (2018, unknown): “representations with image as pivot [22].”
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2021, background): “…in search and retrieval applications, both monolingually for English (Nam et al., 2017; Faghri et al., 2018; Zheng et al., 2020; Wang et al., 2019a; Shi et al., 2019) and in multilingual contexts (Gella et al., 2017; Kádár et al., 2018; Kim et al., 2020; Wehrmann et al., 2019; Burns et al., 2020).”
M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training (2020, unknown): “Table 2 shows the evaluation results on Multi30K and MSCOCO, where M 3 P achieves the state-of-the-art results comparing to several related work [26, 23, 24, 25, 27].”
Models of Visually Grounded Speech Signal Pay Attention to Nouns: A Bilingual Experiment on English and Japanese (2019, unknown): “These corpora, as well as deep learning models, lead to contributions in multilingual language grounding and learning of shared and multimodal representations with neural networks [4, 7, 8, 9, 10, 11, 12, 13].”
Learning to Scale Multilingual Representations for Vision-Language Tasks (2020, background): “ save memory but suer from performance degradation. In contrast, methods that learn to align word embeddings across languages can maintain (or even improve) performance as languages are added (e.g., [16,23]), but require additional parameters for the word embeddings that represent each new language’”
Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting (2020, unknown): “For encoding image and text, we generalize the cross-modality attention from SCAN (Lee et al., 2018) to the multilingual scenario for learning a multilingual VSE space (Gella et al., 2017; Huang et al., 2019a).”
Words Aren’t Enough, Their Order Matters: On the Robustness of Grounding Visual Referring Expressions (2020, background): “Similar proposals are known to increase the robustness of vision and language problems like visual-semantic embeddings and image description ranking (Kiros et al., 2014; Gella et al., 2017; Faghri et al., 2018).”
Learning Visually Grounded Sentence Representations (2017, methodology): “Similar ideas were recently successfully applied to multi-modal machine translation (Elliott and Kádár, 2017; Gella et al., 2017; Lee et al., 2017).”
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021, methodology): “Prior image-text work (Gella et al., 2017; Huang et al., 2019c) utilizes a triplet loss to maintain such neighborhood relationships.”
Emergent Translation in Multi-Agent Communication (2017, background): “It is also related to previous work on multiagent translation for low-resource language pairs (without grounding) (He et al., 2016a) and learning multimodal multilingual representations (Gella et al., 2017).”
Towards Zero-shot Cross-lingual Image Retrieval (2020, methodology): “(Gella et al., 2017) uses images as pivots to perform metric learning which allows text in one language to be independent of the other languages.”
Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech (2018, methodology): “This is similar to the criteria used in [44] for multilingual image/text retrieval, except we randomly sample only a single imposter per (a, p) pair.”
UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training (2021, background): “Naturally, a “Tower of Babel” strategy starts to gain interest in the community, aiming at building one giant model that can handle all languages, notable examples including massively multilingual neural machine translation [1], cross-lingual language model [32], and multilingual multimodal represen”
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages (2022, background): “, 2018), where the perceptual input can be considered as an inherent cross-lingual signal (Kiela et al., 2015; Gella et al., 2017; Caglayan et al., 2021).”
Symbolic Inductive Bias for Visually Grounded Learning of Spoken Language (2018, background): “Gella et al. (2017) and Kádár et al. (2018) learn visual semantic embeddings from textual-visual datasets and show gains from additional languages which reuse the same encoder.”
M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training (2021, methodology): “We compare our work with several related work [15, 31, 11, 32, 1], which are trained on downstream task datasets (MSCOCO and Multi30K) directly without pre-training.”