Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

By Ryan Kiros, Ruslan Salakhutdinov, Richard S. Zemel. 2014

1,463 citations recorded by the source corpus.

Inspired by recent advances in multimodal learning and machine translation, we introduce an encoder-decoder pipeline that learns (a): a multimodal joint embedding space with images and text and (b): a novel language model for decoding distributed representations from our space. Our pipeline effectively unifies joint image-text embedding models with multimodal neural language models. We introduce the structure-content neural language model that disentangles the structure of a sentence to its content, conditioned on representations produced by the encoder. The encoder allows one to rank images and sentences while the decoder can generate novel descriptions from scratch. Using LSTM to encode sentences, we match the state-of-the-art performance on Flickr8K and Flickr30K without using object detections. We also set new best results when using the 19-layer Oxford convolutional network. Furthermore we show that with linear encoders, the learned embedding space captures multimodal regularities in terms of vector space arithmetic e.g. *image of a blue car* - "blue" + "red" is near images of red cars. Sample captions generated for 800 images are made available for comparison.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Over time work explored many combinations of training objective, transfer, and more expressive models and steadily improved performance (Frome et al., 2013; Socher et al., 2014; Karpathy et al., 2014; Kiros et al., 2014; Faghri et al., 2017).”
Root Mean Square Layer Normalization (2019, methodology): “OE treats the caption-image pairs as a two-level partial order, and trains the joint model using the pairwise ranking loss [14].”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “Similar ideas were employed for image captioning [15] and text-based image retrieval [29].”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “Our off-the-shelf HowTo100M model outperforms [22, 24, 53, 64, 65] that are directly trained on MSR-VTT.”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “In this work, we use cosine similarity in the representation space to calculate similarity, which is widely used in learning image-text embedding and shown to be very effective in many prior works [8, 30, 60].”
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018, background): “Automatic image captioning has a long history (Ho-dosh et al., 2013; Donahue et al., 2014; Karpa-thy and Fei-Fei, 2015; Kiros et al., 2015).”
Generative Adversarial Text to Image Synthesis (2016, background): “However, we can still learn an instance level (rather than category level) image and text matching function, as in (Kiros et al., 2014).”
DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis (2019, methodology): “[3] adopted the pair-wise ranking loss [10] to project both images and natural languages into a joint embedding space.”
An Actor-Critic Algorithm for Sequence Prediction (2016, methodology): “This approach has been applied successfully in machine translation (Sutskever et al., 2014; Bahdanau et al., 2015), caption generation (Kiros et al., 2014; Donahue et al., 2015; Vinyals et al., 2015; Xu et al., 2015; Karpathy & Fei-Fei, 2015), and speech recognition (Chorowski et al., 2015; Chan et…”
Grammar-Based Grounded Lexicon Learning (2022, positive): “The concept SHINY is associated with a vector embedding in a joint visual-semantic embedding space, following Kiros et al. [26].”
Coupled Generative Adversarial Networks (2016, methodology): “Our work is related to the prior works in multi-modal learning, including joint embedding space learning [30] and multi-modal Boltzmann machines [1, 3].”
Generative Moment Matching Networks (2015, background): “Kiros et al., 2014), machine translation (Cho et al., 2014; Sutskever et al., 2014), and more.”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “Similar to CLIP and ALIGN, our work also learns the mapping between images and texts, which is related to many previous works, such as (Hironobu et al., 1999; Weston et al., 2010; Socher and Fei-Fei, 2010; Socher et al., 2013; Hodosh et al., 2013; Frome et al., 2013; Norouzi et al., 2013; Kiros et a”