DenseCap: Fully Convolutional Localization Networks for Dense Captioning

By Justin Johnson, Andrej Karpathy, Li Fei-Fei. 2015

1,244 citations recorded by the source corpus.

We introduce the dense captioning task, which requires a computer vision system to both localize and describe salient regions in images in natural language. The dense captioning task generalizes object detection when the descriptions consist of a single word, and Image Captioning when one predicted region covers the full image. To address the localization and description task jointly we propose a Fully Convolutional Localization Network (FCLN) architecture that processes an image with a single, efficient forward pass, requires no external regions proposals, and can be trained end-to-end with a single round of optimization. The architecture is composed of a Convolutional Network, a novel dense localization layer, and Recurrent Neural Network language model that generates the label sequences. We evaluate our network on the Visual Genome dataset, which comprises 94,000 images and 4,100,000 region-grounded captions. We observe both speed and accuracy improvements over baselines based on current state of the art approaches in both generation and retrieval settings.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks (2015, methodology): “Since then, the frameworks of RPN and Faster R-CNN have been adopted and generalized to other methods, such as 3D object detection [13], part-based detection [14], instance segmentation [15], and image captioning [16].”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “These applications include automatic image and video captioning [20, 36, 60, 61], visual question answering [9, 29, 51, 63], visual content retrieval based on textual queries [32, 56, 63], temporal localization of events in videos using natural language [15, 26] or video summarization with natural l”
Deep Perm-Set Net: Learn to predict sets with unknown permutation and cardinality using deep neural networks (2018, background): “…(RNN) neural networks have enjoyed great success in many real-world problems, including scene classification Krizhevsky et al., semantic segmentation Papandreou et al. (2015), speech recognition Hinton et al. (2012), gaming Mnih et al. (2013; 2015), and image captioning Johnson et al. (2016).”
DeepSetNet: Predicting Sets with Deep Neural Networks (2016, background): “Deep neural networks have shown state-of-the-art performance on many computer vision problems, including semantic segmentation [33], visual tracking [31], image captioning [21], scene classification [22], and object detection [27].”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, background): “(Johnson et al., 2016), video-action captioning frameworks (Sun et al., 2019), or scene-graph parsers (Tang et al., 2020).”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “With the advent of deep learning, researchers proposed end-to-end neural networks that encode an image into vector representations and decode a caption word by word [27, 71].”
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering (2017, methodology): “use selective search [45] to identify salient image regions, which are ﬁltered with a classiﬁer then resized and CNNencoded as input to an image captioning model with attention. Similarly to DenseCap [21], Pedersoli et. al. [33] use spatial transformer networks [19] to generate image features, which”
Dense-Captioning Events in Videos (2017, methodology): “Finally, we build upon the recent work on dense-imagecaptioning [16], which generates a set of localized descriptions for an image.”
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization (2016, methodology): “Convolutional Neural Networks (CNNs) and other deep networks have enabled unprecedented breakthroughs in a variety of computer vision tasks, from image classification [24, 16] to object detection [15], semantic segmentation [27], image captioning [43, 6, 12, 21], and more recently, visual question a”
Generation and Comprehension of Unambiguous Object Descriptions (2015, background): “[25, 32] investigate the task of generating dense descriptions in an image.”
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, background): “[16], [17], [7] with a visual feature extractor for conditioning”
What Does Explainable AI Really Mean? A New Conceptualization of Perspectives (2017, background): “Others suggest that rich visualizations or text supplied along with a decision, as is often done in deep learning for computer vision [16,5,6], oﬀer suﬃcient information to draw an explanation of why a particular decision was reached.”
Grad-CAM: Why did you say that? (2016, background): “3b Grad-CAM localizations of a global or holistic captioning model for captions generated by a dense captioning model [3] for the three bounding box proposals marked on the left.”
Pre-Trained Models: Past, Present and Future (2021, background): “…et al., 2014; Gidaris and Komodakis, 2015), image segmentation (Long et al., 2015; Zheng et al., 2015), image caption (Vinyals et al., 2015; Johnson et al., 2016), visual question answering (Antol et al., 2015; Gao et al., 2015; Xiong et al., 2016), etc. Utilizing PTMs like ResNet50 3 has…”
Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning (2017, background): “Despite rapid progress at the intersection of vision and language, in particular, in image/video captioning [4, 14, 36– 38, 41] and question answering [ 2, 25, 28, 34, 35 ] , it is clear we are far from this grand goal of a visual dialog agent.”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, background): “…or visually-grounded question-answer pairs (Kiros et al., 2014; Shi et al., 2018; Mao et al., 2016; Vendrov et al., 2016; Ganju et al., 2017), dense language descriptions for scenes (Johnson et al., 2016), video-captioning (Donahue et al., 2015) and video-text alignment (Zhu et al., 2015).”
Modeling Context in Referring Expressions (2016, background): “For the referring expression generation task, rather than generating sentences for each object in an image separately [15][26], we consider tying the generation process together into a single task to jointly generate expressions for all objects of the same object category depicted in an image.”