Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge

By Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan. 2016

915 citations recorded by the source corpus.

Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. Finally, given the recent surge of interest in this task, a competition was organized in 2015 using the newly released COCO dataset. We describe and analyze the various improvements we applied to our own baseline and show the resulting performance in the competition, which we won ex-aequo with a team from Microsoft Research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Learning Visual N-Grams from Web Data (2016, methodology): “ (left) and image retrieval (right) performance of four baseline models and our visual n -gram models on COCO-1K. We do not reportresultsweobtainedwiththelastversionoftheneural image captioning model [4] here because that model was trained on COCO validation set that was used as the basis for the CO”
wav2vec: Unsupervised Pre-training for Speech Recognition (2019, neutral): “In computer vision, representations for ImageNet (Deng et al., 2009) and COCO (Lin et al., 2014) have proven to be useful to initialize models for tasks such as image captioning (Vinyals et al., 2016) or pose estimation (Pavllo et al., 2019).”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, background): “Joint visual-text models have become increasingly popular as they enable a wide suite of downstream tasks, including text-to-visual retrieval [29, 31, 36, 59], visual captioning [24, 58, 66], and visual question and answering [4, 27].”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “With the advent of deep learning, researchers proposed end-to-end neural networks that encode an image into vector representations and decode a caption word by word [27, 71].”
Self-Critical Sequence Training for Image Captioning (2016, methodology): “We outperform the previous best system on all evaluation metrics.”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, methodology): “A common approach for these tasks is to use temporal models over convolutional features [36, 70, 7].”
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, methodology): “In particular, we compare with: FC-2K [20], an LSTM baseline using a global feature vector as image descriptor; Att2all [20], which uses additive attention over the grid of image features extracted from a CNN; Up-Down [8], which employs a two-LSTM layer model with bottom-up features extracted from F”
Evolving Deep Neural Networks (2017, background): “performing beam search or scheduled sampling during training [48].”
Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates (2017, background): “Recurrent neural networks (RNNs) which can handle the sequential data with variable lengths [13], [14], have shown their strength in language modeling [15], [16], [17], image captioning [18], [19], video analysis [20], [21], [22], [23], [24],”
Meshed-Memory Transformer for Image Captioning (2019, methodology): “With the advent of Deep Neural Networks, most captioning techniques have employed RNNs as language models and used the output of one or more layers of a CNN to encode visual information and condition language generation [43, 33, 9, 16].”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, background): “Generating well-formed sentences requires both syntactic and semantic understanding of the language [143].”
Visual Entailment Task for Visually-Grounded Language Learning (2018, methodology): “Image Captioning: Before VE, there are many captioning models [Karpathy and Fei-Fei, 2015, Vinyals et al., 2017, Chen et al., 2017] which can serve as a useful baseline by generating an image caption as the premise and then apply existing TE models for classification.”
Semantic Object Accuracy for Generative Text-to-Image Synthesis (2019, methodology): “To this end, [31] use a pre-trained caption generator [57] to generate captions for each synthesized image and compare these to the original ones through standard language similarity metrics, i.”
Neural Learning of One-of-Many Solutions for Combinatorial Problems in Structured Output Spaces (2020, background): “…other tasks such as Neural Machine translation [Bahdanau et al., 2015, Sutskever et al., 2014], Summarization [Nallapati et al., 2017, Paulus et al., 2018], Image Captioning [Vinyals et al., 2017, You et al., 2016] etc., where one would expect to have multiple valid solutions for any given input.”
Eigen-CAM: Class Activation Map using Principal Components (2020, methodology): “The widely used convolutional neural network (CNN ) based architectures in DL are proven to be the best in learning representations and solving complex computer vision and general artificial intelligence problems such as image classification [4], [5], object localization [6]–[8], semantic segmentati”
Semantic bottleneck for computer vision tasks (2018, methodology): “rature. However, with the exception of the DAP model [7], which is not performing very well, such models produce vector representations that are not intelligible at all. In contrast, image captioning [8,9] is by nature producing intelligible representations and can be used to index images. As an ill”
RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words (2021, methodology): “The main development of image captioning [41, 40, 5, 14, 24] can be divided into two stages: traditional method stage and deep learning method stage.”
Radiology Objects in COntext (ROCO): A Multimodal Image Dataset (2018, methodology): “Generative models trained on ROCO image - caption pairs can be used to automatically create natural sentences describing radiology images, as proposed in [14, 23].”