Deep Visual-Semantic Alignments for Generating Image Descriptions

By Andrej Karpathy, Li Fei-Fei. 2014

6,086 citations recorded by the source corpus.

We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks (RNN) over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions outperform retrieval baselines on both full images and on a new dataset of region-level annotations. Finally, we conduct large-scale analysis of our RNN language model on the Visual Genome dataset of 4.1 million captions and highlight the differences between image and region-level caption statistics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Yoshua Bengio
  4. Percy Liang
  5. Quoc V. Le
  6. Luke S. Zettlemoyer
  7. Trevor Darrell
  8. Caiming Xiong

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Jitendra Malik
  7. Yi Yang
  8. Andrew Zisserman

In Natural Language Processing

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Richard Socher
  4. Yejin Choi
  5. Graham Neubig
  6. Luke S. Zettlemoyer
  7. Kyunghyun Cho
  8. Caiming Xiong

In Multimodal Learning

  1. Zhe Gan
  2. Devi Parikh
  3. Mohit Bansal
  4. Jianfeng Gao
  5. Marcus Rohrbach
  6. Qi Wu
  7. Linjie Li
  8. Lijuan Wang

Citations in the corpus