Joint Image-Text Representation by Gaussian Visual-Semantic Embedding

By Zhou Ren, Hailin Jin, Zhe L. Lin, Chen Fang, Alan L. Yuille. 2016

54 citations recorded by the source corpus.

How to jointly represent images and texts is important for tasks involving both modalities. Visual-semantic embedding models have been recently proposed and shown to be effective. The key idea is that by learning a mapping from images into a semantic text space, the algorithm is able to learn a compact and effective joint representation. However, existing approaches simply map each text concept to a single point in the semantic space. Mapping instead to a density distribution provides many interesting advantages, including better capturing uncertainty about each text concept, and enabling better geometric interpretation of concepts such as inclusion, intersection, etc. In this work, we present a novel Gaussian Visual-Semantic Embedding (GVSE) model, which leverages the visual information to model text concepts as Gaussian distributions in semantic space. Experiments in two tasks, image classification and text-based image retrieval on the large scale MIT Places205 dataset, have demonstrated the superiority of our method over existing approaches, with higher accuracy and better robustness.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “on space using kernelized canonical correlation analysis, and achieve good results for annotation and segmentation. Similar ideas are employed for image captioning [14] and text-based image retrieval [30]. In particular, the seminal work DeViSE [8] proposes to identify visual objects using semantic ”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “Visual semantic embedding [117, 118] is used to compute the actual reward value in predicting the correct word.”
A Survey of Zero-Shot Learning (2019, background): “In addition to generating one prototype for each class, there are also works [103, 125] that generate more than one prototype for each class in the label embedding space.”
Deep Reinforcement Learning-Based Image Captioning with Embedding Reward (2017, methodology): “In this paper, we propose to train using an actor-critic model [21] with reward driven by visualsemantic embedding [11, 19, 36, 37].”
Learning Social Image Embedding with Deep Multimodal Attention Networks (2017, methodology): “[23] proposes a Gaussian visual-semantic embedding model for joint image-text representation, which leverages the visual information to model text concepts as Gaussian distributions in semantic space.”
SibNet: Sibling Convolutional Encoder for Video Captioning (2018, background): “The success of visual-semantic joint embedding in image classification [48], [49] and image retrieval [50], [51] inspires us to implement our semantic branch via visual-semantic joint embedding.”
Universal Multimodal Representation for Language Understanding (2023, background): “The major focus is to bridge the gap between text and images through building visualsemantic embeddings [39], [40], [41], [42], [43].”
Generative Models of Visually Grounded Imagination (2017, background): “However, there are also a few that use Gaussian embeddings for words (Vilnis and McCallum, 2015; Athiwaratkun and Wilson, 2017), sometimes in conjunction with images (Mukherjee and Hospedales, 2016; Ren et al., 2016).”
Unified Visual-Semantic Embeddings: Bridging Vision and Language With Structured Meaning Representations (2019, methodology): “Word-level GRU GRU Word-level GRU Word-level A fundamental technique proposed in [13] for aligning two modalities is to use the pairwise ranking to learn a distance metric from similar and dissimilar cross-modal pairs [44, 35, 23, 9, 28, 24].”
Using Image Fairness Representations in Diversity-Based Re-ranking for Recommendations (2018, methodology): “We are also interested in exploring whether our method works for fairness representations that are computed from word embedding methods [7] and joint image-text embedding methods [12].”