From captions to visual concepts and back

By Hao Fang, Saurabh Gupta, Forrest N. Iandola, Rupesh Kumar Srivastava, Li Deng, Piotr Doll'ar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. Platt and 2 more. 2014

1,357 citations recorded by the source corpus.

This paper presents a novel approach for automatically generating image descriptions: visual detectors, language models, and multimodal similarity models learnt directly from a dataset of image captions. We use multiple instance learning to train visual detectors for words that commonly occur in captions, including many different parts of speech such as nouns, verbs, and adjectives. The word detector outputs serve as conditional inputs to a maximum-entropy language model. The language model learns from a set of over 400,000 image descriptions to capture the statistics of word usage. We capture global semantics by re-ranking caption candidates using sentence-level features and a deep multimodal similarity model. Our system is state-of-the-art on the official Microsoft COCO benchmark, producing a BLEU-4 score of 29.1%. When human judges compare the system captions to ones written by other people on our held-out test set, the system captions have equal or better quality 34% of the time.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Learning Visual N-Grams from Web Data (2016, background): “Most image-captioning systems train a recurrent network or maximum entropy language model on top of object classifications produced by a convolutional network; the models are either trained separately [14, 29, 43] or end-to-end [15, 59].”
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations (2016, background): “Among those in the second category, a common theme has been to use recurrent neural networks to produce novel captions (Kiros et al. 2014; Mao et al. 2014; Karpathy and Fei-Fei 2015;Vinyals et al. 2015;Chen andLawrenceZitnick 2015; Donahue et al. 2015; Fang et al. 2015).”
Object-Centric Image Generation from Layouts (2020, methodology): “There has been strong interest in image and caption similarity modules for retrieval (Fang et al. 2015; Huang et al. 2013) and for text-to-image generation, most recently with the DAMSM model proposed in (Xu et al.”
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, methodology): “Following [4, 9], we define the loss function as the negative log posterior probability that the images are matched with their corresponding text descriptions (ground truth), i.”
STaR: Bootstrapping Reasoning With Reasoning (2022, background): “The domains need not be constrained to language either – tasks leveraging language models in other domains are a natural extension such as visual question answering [Fang et al., 2015] and language-guided reinforcement learning [Mu et al.”
ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “Early works utilize the features extracted from a pre-trained classification network [6, 9, 13, 42], while later works [4, 19, 47] exploit the more expressive features of an object detection network [31].”
Attention on Attention for Image Captioning (2019, background): “It requires not only to recognize salient objects in an image, understand their interactions, but also to verbalize them using natural language, which makes itself very challenging [25, 45, 28, 12].”
Visual Dialog (2016, background): “A number of problems at the intersection of vision and language have recently gained prominence – image captioning [10, 11, 20, 54], video/movie description [43, 51, 52], text-to-image coreference/grounding [16, 22, 37, 39, 42], Visual Madlibs [60], and of course, visual question answering (VQA) [4,”
VQA: Visual Question Answering (2015, methodology): “In particular, research in image and video captioning that combines Computer Vision (CV), Natural Language Processing (NLP), and Knowledge Representation & Reasoning (KR) has dramatically increased in the past year [13, 7, 9, 32, 21, 19, 45].”
Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “Recently there has been a surprising resurgence of interest in this area [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], due to the renewed interest in neural network learning techniques [31], [32] and increasingly large datasets [33], [34], [35], [7], [36], [37], [38].”
Long-term recurrent convolutional networks for visual recognition and description (2014, neutral): “2 [39] MSR Captivator 0.907 0.819 0.710 0.601 0.937 0.339 0.680 [40] m-RNN (2015) 0.890 0.798 0.687 0.575 0.935 0.325 0.666 (Ours) * LRCN, this work (sample) 0.895 0.804 0.695 0.585 0.934 0.335 0.678 [41] MSR 0.880 0.789 0.678 0.567 0.925 0.331 0.662 [42] Nearest Neighbor 0.872 0.770 0.655 0.542 0.9”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “More closely related to our approach is the work of Srivas-tava et al. [43] who use a Deep Boltzmann Machine to learn a joint distribution over a images and tags.”
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (2015, background): “COCO CMU/MS Research (Chen & Zitnick, 2014) MS Research (Fang et al., 2014)†a BRNN (Karpathy & Li, 2014)◦ Google NIC†◦Σ Log Bilinear◦ Soft-Attention Hard-Attention — — 64.”
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering (2017, neutral): “LEU-4 METEOR ROUGE-L CIDEr SPICE c5 c40 c5 c40 c5 c40 c5 c40 c5 c40 c5 c40 c5 c40 c5 c40 Google NIC [48] 71.3 89.5 54.2 80.2 40.7 69.4 30.9 58.7 25.4 34.6 53.0 68.2 94.3 94.6 18.2 63.6 MSR Captivator [11] 71.5 90.7 54.3 81.9 40.7 71.0 30.8 60.1 24.8 33.9 52.6 68.0 93.1 93.7 18.0 60.9 M-RNN [31] 71.6”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, methodology): “…a recent surge of work in this area, and in particular, on the task of sentence-based image description (Chen and Zitnick 2015; Donahue et al. 2015; Fang et al. 2015; Farhadi et al. 2010; Hodosh et al. 2013; Karpathy and Fei-Fei 2015; Kiros et al. 2014; Klein et al. 2014; Kulkarni et al. 2011;…”
SPICE: Semantic Propositional Image Caption Evaluation (2016, background): “As shown in Table 2, the MSR entry [38] —incorporating speciﬁcally trained visual detectors for nouns, verbs and adjectives—exceeds the human F-score baseline for tuples containing color attributes.”