Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images

By Junhua Mao, Jiajing Xu, Yushi Jing, Alan L. Yuille. 2016

51 citations recorded by the source corpus.

In this paper, we focus on training and evaluating effective word embeddings with both text and visual information. More specifically, we introduce a large-scale dataset with 300 million sentences describing over 40 million images crawled and downloaded from publicly available Pins (i.e. an image with sentence descriptions uploaded by users) on Pinterest. This dataset is more than 200 times larger than MS COCO, the standard large-scale image dataset with sentence descriptions. In addition, we construct an evaluation dataset to directly assess the effectiveness of word embeddings in terms of finding semantically similar or related words and phrases. The word/phrase pairs in this evaluation dataset are collected from the click data with millions of users in an image search system, thus contain rich semantic relationships. Based on these datasets, we propose and compare several Recurrent Neural Networks (RNNs) based multimodal (text and image) models. Experiments show that our model benefits from incorporating the visual information into the word embeddings, and a weight sharing strategy is crucial for learning such multimodal embeddings. The project page is: http://www.stat.ucla.edu/~junhua.mao/multimodal_embedding.html (The datasets introduced in this work will be gradually released on the project page.).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018, methodology): “Also related to this work is the Pinterest image and sentence-description dataset (Mao et al., 2016).”
nocaps: novel object captioning at scale (2019, methodology): “In contrast, filtered datasets, such as Im2Text [27], Pinterest40M [26] and Conceptual Captions [36], contain large numbers of image-caption pairs harvested from the web.”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, background): “…concepts from descriptive languages, such as image-captioning or visually-grounded question-answer pairs (Kiros et al., 2014; Shi et al., 2018; Mao et al., 2016; Vendrov et al., 2016; Ganju et al., 2017), dense language descriptions for scenes (Johnson et al., 2016), video-captioning (Donahue…”
Deep Multimodal Learning: A Survey on Recent Advances and Trends (2017, unknown): “[99] 2016”
Learning Visually-Grounded Semantics from Contrastive Adversarial Samples (2018, background): “016). These embeddings map multiple domains into a joint vector space which describes the semantical relations between inputs (e.g., distance, correlation). We focus on the visual-semantic embedding (Mao et al., 2016; Kiros et al., 2014; Faghri et al., 2017), learning word embeddings with visually-g”
Related Pins at Pinterest: The Evolution of a Real-World Recommender System (2017, methodology): “…metadata Image features : fc6 and fc8 activations of a deep convolutional neural network [15] Word embeddings : aggregated for the text annotations [20] Category vector : aggregated from user-selected categories for their boards Topic vectors : computed from the pin-board graph Demographic data :…”
Illustrative Language Understanding: Large-Scale Visual Grounding with Image Search (2018, background): “…tasks. also fused text-based representations with image-based representations (Bruni et al., 2014; Lazari-dou et al., 2015; Chrupala et al., 2015; Mao et al., 2016; Silberer et al., 2017; Kiela et al., 2017; Collell et al., 2017; Zablocki et al., 2018) and representations derived from a…”
Multimodal Sentiment Analysis: A Survey of Methods, Trends, and Challenges (2023, background): “[147] focused on training and evaluating effective word embeddings with text and visual information.”