Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics

By Douwe Kiela, L. Eon Bottou. 2014

246 citations recorded by the source corpus.

We construct multi-modal concept representations by concatenating a skip-gram linguistic representation vector with a visual concept representation vector computed using the feature extraction layers of a deep convolutional neural network (CNN) trained on a large labeled object recognition dataset. This transfer learning approach brings a clear performance gain over features based on the traditional bag-of-visual-word approach. Experimental results are reported on the WordSim353 and MEN semantic relatedness evaluation tasks. We use visual features computed using either ImageNet or ESP Game images.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Visual Features from Large Weakly Supervised Data (2015, methodology): “Following [24] and others, we measure the quality of word embeddings by the Spearman’s rank correlation of the cosine similarity of the word pairs and the human-provided similarity rating for those pairs.”
Learning to Answer Questions from Image Using Convolutional Neural Network (2015, background): “Recently, there is an increasing research area for the multimodal learning between image and language (Kiela et al., 2014; Gong et al., 2014; Sri-vastava et al., 2012a; Srivastava et al., 2012b).”
Combining Language and Vision with a Multimodal Skip-gram Model (2015, methodology): “We test the vectors made available by Kiela and Bottou (2014).”
Cross-Modal and Hierarchical Modeling of Video and Text (2018, background): “Examples are embedding images and the visual categories [8], embedding images and texts for VSE [19], and embedding images, questions, and answers for VQA [13].”
Associative Multichannel Autoencoder for Multimodal Word Representation (2018, methodology): “The Concatenation (CONC) model (Kiela and Bottou, 2014) is simple concatenation of normalized textual and visual vectors.”
Tensor Networks : Deep Learning and Logical Reasoning from Data and Knowledge ⋆ (2016, background): “The recent availability of large-scale data combining mult iple data modalities, such as image, text, audio and sensor data, has opened up various r esearch and commercial opportunities, underpinned by machine learning metho ds and techniques [5, 12, 17, 18].”
Multimodal Machine Learning: A Survey and Taxonomy (2017, methodology): “Starting from work by Feng and Lapata [62], grounding is usually performed by finding a common latent space between the representations [62], [190] (in case of parallel datasets) or by learning unimodal representations separately and then concatenating them to lead to a multimodal one [30], [105], [”
A Survey of Multi-View Representation Learning (2016, unknown): “For example, Kiela and Bottou [33] obtain multi-modal concept representations by concatenating a skip-gram linguistic representation vector with a visual concept representation learned with a deep convolution neural network.”
Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images (2016, methodology): “One type of methods takes a two-step strategy that ﬁrst extracts text and image features separately and then fuses them together using singular value decomposition [5], stacked autoencoders [31], or even simple concatenation [17].”
Gated Multimodal Units for Information Fusion (2017, methodology): “A simple approach is to concatenate features to get a ﬁnal representation (Kiela & Bottou, 2014; Pei et al., 2013; Suk & Shen, 2013).”
Multimodal Classification Fusion in Real-World Scenarios (2017, background): “Some examples of early fusion such as [10], [6], directly concatenate text and image features to produce a single multimodal vector (see graphical representation in Figure 3), obtaining promising performance in other contexts other than classification.”
Illustrative Language Understanding: Large-Scale Visual Grounding with Image Search (2018, methodology): “There has also been other work using other image sources such as ImageNet (Kiela and Bottou, 2014; Collell and Moens, 2016) over the WordNet synset vocabulary, and using Flickr photos and captions (Joulin et al., 2016).”
Monolingual and Cross-Lingual Information Retrieval Models Based on (Bilingual) Word Embeddings (2015, background): “(3), the model incrementally pushes observed pivot WEs towards context WEs of their collocates in the corpus.”
Learning Visually Grounded Sentence Representations (2017, background): “Concatenation has been proven to be a strong and straightforward mid-level multi-modal fusion method, previously explored in multi-modal semantics for word representations (Bruni et al., 2014; Kiela and Bottou, 2014).”
Sentiment Analysis: Detecting Valence, Emotions, and Other Affectual States from Text (2016, background): “• Multimodal Datasets: There is new work on developing word representations not only from text corpora but also from collections of images that the words are associated with (Kiela & Bottou, 2014; Hill & Korhonen, 2014; Lazaridou, Bruni, & Baroni, 2014). (An image and a word can be considered associ”
Aligning Cross-Lingual Entities with Multi-Aspect Information (2019, background): “Such fusion through concatenation is also known as Scoring Level Fusion, which has been proven simple but effective for capturing multimodal semantics (Bruni et al., 2014; Kiela and Bottou, 2014; Collell et al., 2017).”
Quantifying the Visual Concreteness of Words and Topics in Multimodal Datasets (2018, background): “Other work has quantiﬁed characteristics of concrete-ness in multimodal datasets (Young et al., 2014; Hill et al., 2014; Hill and Korhonen, 2014b; Kiela and Bottou, 2014; Jas and Parikh, 2015; Lazari-dou et al., 2015; Silberer et al., 2016; Lu et al., 2017; Bhaskar et al., 2017).”