Learning a Recurrent Visual Representation for Image Caption Generation

By Xinlei Chen, C. Lawrence Zitnick. 2014

203 citations recorded by the source corpus.

In this paper we explore the bi-directional mapping between images and their sentence-based descriptions. We propose learning this mapping using a recurrent neural network. Unlike previous approaches that map both sentences and images to a common embedding, we enable the generation of novel sentences given an image. Using the same model, we can also reconstruct the visual features associated with an image given its visual description. We use a novel recurrent visual memory that automatically learns to remember long-term visual concepts to aid in both sentence generation and visual feature reconstruction. We evaluate our approach on several tasks. These include sentence generation, sentence retrieval and image retrieval. State-of-the-art results are shown for the task of generating novel image descriptions. When compared to human generated captions, our automatically generated captions are preferred by humans over $19.8\%$ of the time. Results are better than or comparable to state-of-the-art results on the image and sentence retrieval tasks for methods using similar visual features.

Subfields: Computer Vision and Pattern Recognition, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Trevor Darrell
  3. Dacheng Tao
  4. Xiaogang Wang
  5. Yi Yang
  6. Jitendra Malik
  7. Chunhua Shen
  8. Ming-Hsuan Yang

In Natural Language Generation

  1. Marcus Rohrbach
  2. Trevor Darrell
  3. Kate Saenko
  4. Devi Parikh
  5. Anna Rohrbach
  6. Li Fei-Fei
  7. Qi Wu
  8. Subhashini Venugopalan

Citations in the corpus