Combining Language and Vision with a Multimodal Skip-gram Model

By Angeliki Lazaridou, Ngoc-Quan Pham, Marco Baroni. 2015

290 citations recorded by the source corpus.

We extend the SKIP-GRAM model of Mikolov et al. (2013a) by taking visual information into account. Like SKIP-GRAM, our multimodal models (MMSKIP-GRAM) build vector-based word representations by learning to predict linguistic contexts in text corpora. However, for a restricted set of words, the models are also exposed to visual representations of the objects they denote (extracted from natural images), and must predict linguistic and visual features jointly. The MMSKIP-GRAM models achieve good performance on a variety of semantic benchmarks. Moreover, since they propagate visual information to all words, we use them to improve image labeling and retrieval in the zero-shot setup, where the test concepts are never seen during model training. Finally, the MMSKIP-GRAM models discover intriguing visual properties of abstract words, paving the way to realistic implementations of embodied theories of meaning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “Recently there has been a surprising resurgence of interest in this area [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], due to the renewed interest in neural network learning techniques [31], [32] and increasingly large datasets [33], [34], [35], [7], [36], [37], [38].”
Exploring Nearest Neighbor Approaches for Image Captioning (2015, background): “We hope to provide context for the recent advances in this area [18], [26], [35], [16], [19], [6], [8], [2], [23], [22], [21], [36].”
Bayesian Neural Word Embedding (2016, background): “Specifically, Skip-Gram (SG) with negative sampling, known also as word2vec [4], set new records in various linguistic tasks and its applications have been extended to other domains beyond NLP such as computer vision [5], [6], Collaborative Filtering (CF) and item similarity [7].”
Supervised Multimodal Bitransformers for Classifying Images and Text (2019, background): “While concatenation can be considered the default, other fusion meth-ods have been explored e.g. for lexical representation learning (Bruni, Tran, and Baroni 2014; Lazaridou, Pham, and Baroni 2015).”
Are Distributional Representations Ready for the Real World? Evaluating Word Vectors for Grounded Perceptual Meaning (2017, neutral): “Despite the success of distributional representations in standard natural language processing tasks, a small but growing consensus within the artificial intelligence community suggests that these methods cannot be sufficient to induce adequate representations of words and concepts (Kiela et al., 201”
“Look, some Green Circles!”: Learning to Quantify from Images (2016, background): “…shown, in particular, that the meaning of content words can be modelled in a cognitively – and even neurosci-entiﬁcally – plausible way by learning representations from both the linguistic and visual contexts in which a lexical item has been observed (Anderson et al., 2013; Lazaridou et al., 2015).”
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities (2018, unknown): “Earlier work used fusion approaches such as concatenation of input features (Ngiam et al. 2011; Lazaridou, Pham, and Baroni 2015).”
From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning (2016, methodology): “Some recent models have used more naturalistic, larger-scale inputs, for example in cross-modal distributional semantics (Lazaridou et al., 2015) or in implementations of the acquisition process trained on images paired with their descriptions (Chrupała et al.”
Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images (2016, background): “Some of these models directly minimize the Euclidean distance between the visual features and the word embeddings or RNN states, similar to previous work (e.g. [13, 21]).”
Factorized Multimodal Transformer for Multimodal Sequential Learning (2019, methodology): “Hybrid methods have also been used to combine early and late fusion together (Wu et al., 2019; Nguyen & Okatani, 2018; Lazaridou et al., 2015; Hu et al., 2017).”
Automatic Spatially-Aware Fashion Concept Discovery (2017, background): “Our work is also related to visual-semantic embedding models [5, 10, 31, 14, 7].”
Quantifying the Visual Concreteness of Words and Topics in Multimodal Datasets (2018, background): “2014a). Other work has quantiﬁed characteristics of concreteness in multimodal datasets (Young et al., 2014; Hill et al., 2014; Hill and Korhonen, 2014b; Kiela and Bottou, 2014; Jas and Parikh, 2015; Lazaridou et al., 2015; Silberer et al., 2016; Lu et al., 2017; Bhaskar et al., 2017). Most related ”
Illustrative Language Understanding: Large-Scale Visual Grounding with Image Search (2018, background): “also fused text-based representations with imagebased representations (Bruni et al., 2014; Lazaridou et al., 2015; Chrupala et al., 2015; Mao et al., 2016; Silberer et al., 2017; Kiela et al., 2017; Collell et al., 2017; Zablocki et al., 2018) and representations derived from a knowledge-graph (Thom”
Multilingual Visual Sentiment Concept Matching (2016, unknown): “visual similarity to evaluate word embeddings [13, 14].”
Deep Multimodal Representation Learning: A Survey (2019, result): “[35] A. Lazaridou and M. Baroni, ‘‘Combining language and vision with a multimodal skip-gram model,’’ in Proc.”
The Importance of Modeling Social Factors of Language: Theory and Practice (2021, unknown): “(Lazaridou et al., 2015; Caglayan et al., 2019).”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “Similarly, the distances between word embeddings can be modified according to the similarities between their visual instantiations [115], which are deter-”