Language Models for Image Captioning: The Quirks and What Works

By Jacob Devlin, Hao Cheng, Hao Fang, Saurabh Gupta, Li Deng, Xiaodong He, Geoffry Zweig, Margaret Mitchell. 2015

293 citations recorded by the source corpus.

Two recent approaches have achieved state-of-the-art results in image captioning. The first uses a pipelined process where a set of candidate words is generated by a convolutional neural network (CNN) trained on images, and then a maximum entropy (ME) language model is used to arrange these words into a coherent sentence. The second uses the penultimate activation layer of the CNN as input to a recurrent neural network (RNN) that then generates the caption sequence. In this paper, we compare the merits of these different language modeling approaches for the first time by using the same state-ofthe-art CNN as input. We examine issues in the different approaches, including linguistic irregularities, caption repetition, and data set overlap. By combining key aspects of the ME and RNN methods, we achieve a new record performance over previously published results on the benchmark COCO dataset. However, the gains we see in BLEU do not translate to human judgments.

Subfields: Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Large Language Models

Citations in the corpus

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, background): “o reproduce generic captions from the training data, and to perform poorly on compositionally novel images whose objects may have been seen individually at training time, but not in that combination (Devlin et al. 2015). Some recent works do try to ﬁnd correspondences between image regions and parts”
Stacked Cross Attention for Image-Text Matching (2018, background): “A rich line of studies have explored mapping whole images and full sentences to a common semantic vector space for image-text matching [22, 38, 2, 39, 23, 27, 44, 10, 33, 13, 9, 8, 11].”
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN) (2014, methodology): “We compare our model with several state-of-the-art methods: SDT-RNN (Socher et al. (2014)), DeViSE (Frome et al. (2013)), DeepFE (Karpathy et al. (2014)) with various image representations.”
Movie Description (2016, methodology): “As Hagrid carries Malfoy away, the hippogriﬀ gen-tly nudges Harry.”
Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering (2015, methodology): “Most methods use the soft attention mechanism first proposed in [21], which adds a layer to the network that predicts soft weights and uses them to compute a weighted combination of the items in memory.”
Exploring Nearest Neighbor Approaches for Image Captioning (2015, methodology): “In our approach [5] we ﬁrst ﬁnd a set of k NN images in the training dataset.”
Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge (2016, methodology): “An “encoder” RNN reads the source sentence and transforms it into a rich ﬁxed-length vector representation, which in turn in used as the initial hidden state of a “decoder” RNN that generates the target sentence.”
Describing Multimedia Content Using Attention-Based Encoder-Decoder Networks (2015, neutral): “[20], ( • ) [18], ( ◦ ) [45], ( (cid:5) ) [46] AND ( ∗ ) [22].”
Guided Open Vocabulary Image Captioning with Constrained Beam Search (2016, methodology): “Our approach to out-of-domain image captioning could be applied to any existing CNN-RNN captioning model that can be decoding using beam search, e.g., (Donahue et al., 2015; Mao et al., 2015; Karpathy and Fei-Fei, 2015; Vinyals et al., 2015; Devlin et al., 2015).”
Summarizing Source Code using a Neural Attention Model (2016, background): “, 2011), image and video description (Karpathy and Li, 2015; Venugopalan et al., 2015; Devlin et al., 2015), sentence summarization (Rush et al.”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, methodology): “For example, both Yagcioglu et al. (2015) and (Devlin et al., 2015b) retrieve and rank captions for a query image, using a cnn for the representation of the visual space.”
Semantic Compositional Networks for Visual Captioning (2016, background): “Representative works include [7, 9, 10, 20, 23, 24, 29, 48] for image captioning and [10, 46, 47, 56, 3, 35, 11] for video captioning.”
What Value Do Explicit High Level Concepts Have in Vision to Language Problems? (2015, neutral): “. Xu et al. [53] proposed a model based on visual attention, as well as You et al. [56]. Jia et al. [18] applied additional retrieved sentences to guide the LSTM in generating captions. Devlin et al. [9] combined both maximum entropy (ME) language model and RNN to generate captions. Interestingly, t”
Boosting Image Captioning with Attributes (2016, methodology): “The research on image captioning has proceeded along three different dimensions: template-based methods [11, 18, 34], search-based approaches [4, 7], and languagebased models [5, 16, 31, 32, 33, 36, 38].”
Towards Diverse and Natural Image Descriptions via a Conditional GAN (2017, background): “As illustrated in Figure 2, when similar images are presented, the sentences generated by such a model often contain repeated patterns [3].”