A Hierarchical Approach for Generating Descriptive Image Paragraphs

By Jonathan Krause, Justin Johnson, Ranjay Krishna, Li Fei-Fei. Computer Vision and Pattern Recognition, 2016

410 citations recorded by the source corpus.

Recent progress on image captioning has made it possible to generate novel sentences describing images in natural language, but compressing an image into a single sentence can describe visual content in only coarse detail. While one new captioning approach, dense captioning, can potentially describe images in finer levels of detail by captioning many regions within an image, it in turn is unable to produce a coherent story for an image. In this paper we overcome these limitations by generating entire paragraphs for describing images, which can tell detailed, unified stories. We develop a model that decomposes both images and paragraphs into their constituent parts, detecting semantic regions in images and using a hierarchical recurrent neural network to reason about language. Linguistic analysis confirms the complexity of the paragraph generation task, and thorough experiments on a new dataset of image and paragraph pairs demonstrate the effectiveness of our approach.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3a7011346ce939e3251915e92ae2f252e4c7f777

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Connecting Vision and Language with Localized Narratives (2019, background): “2 Stanford Visual Paragraphs [30] 61.”
A Dataset for Telling the Stories of Social Media Videos (2018, background): “Other multi-sentence description datasets are proposed for story narration of sets of images taken from a Flickr album (Huang et al., 2016; Krause et al., 2017).”
GaitSet: Regarding Gait as a Set for Cross-View Gait Recognition (2018, methodology): “Recently, such methods are introduced into computer vision domains like content recommendation (Hamilton, Ying, and Leskovec 2017) and image cap-tioning (Krause et al. 2017) to aggregate features in a form of a set.”
Towards Diverse and Natural Image Descriptions via a Conditional GAN (2017, methodology): “Specifically, our extended design is inspired by [11].”
Learning to Write with Cooperative Discriminators (2018, background): “Generation with Long-term Context Several prior works studied paragraph generation using sequence-to-sequence models for image captions (Krause et al., 2017), product reviews (Lipton et al., 2015; Dong et al., 2017), sport reports (Wiseman et al., 2017), and recipes (Kiddon et al., 2016).”
Encoder-Agnostic Adaptation for Conditional Language Generation (2019, methodology): “xperiments use the GPT2 BPE scheme, so PPL numbers are not directly comparable to those reported in (Fan et al., 2018)). Bold indicates statistically signiﬁcant best results at p0:05. Model CIDEr B4 Krause et al. (2017) 13.5 8.7 Chatterjee et al. (2018) 20.9 9.4 Melas-Kyriazi et al. (2018) 22.7 8.7 ”
Recurrent Topic-Transition GAN for Visual Paragraph Generation (2017, methodology): “In terms of the fully-supervised setting, to make a fair comparison with the state-of-the-art methods [13, 16], the experiments are conducted on the public image paragraph dataset [16], where 14,575 image-paragraph pairs are used for training, 2,487 for validation and 2,489 for testing.”
Teaching Machines to Describe Images via Natural Language Feedback (2017, methodology): “In [14], the authors use a two-level LSTM to generate paragraphs, while [34] uses it to generate sentences as a sequence of phrases.”
Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports (2019, background): “Some other efforts have been made for building variants of the hierarchical Long-Short-Term-Memory (LSTM) network (Hochreiter and Schmidhuber, 1997) to generate paragraphs (Krause et al., 2017; Yu et al., 2016; Liang et al., 2017).”
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks (2023, background): “…approach incorporates three distinct learning objectives, each addressing a different level of granularity and semantic understanding: • Image-level understanding tasks capture high-level semantics and foster a comprehensive understanding of images through linguistic descriptions [13, 18, 34, 91].”
Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation (2021, methodology): “The Base model is the implementations of HRNN [19] models.”
Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention (2017, result): “Differently from previous works which aim at predicting a single caption, Krause et al. [24] introduced the generation of entire paragraphs for describing images.”
3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans (2020, methodology): “have been used for image retrieval [35], captioning [3, 36, 46], high-level understanding [17, 32, 34, 112], visual question-”
Dense-Captioning Events in Videos (2017, methodology): “Similar to previous work on language modeling [20, 18], we use a cross-entropy loss across all words in every sentence.”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, methodology): “We evaluate the multimodal vision and language capabilities of D REAM LLM across several benchmarks, including image-to-text captioning on COCO (Karpathy & Fei-Fei, 2017) and Image2Paragraph (Krause et al., 2017), general visual question answering (VQA) on VQAv2 (Goyal et al., 2019), OKVQA (Marino e”
Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning (2020, background): “, 2017)), or for generating paragraphs or multisentence captions about the image or video (Krause et al., 2017; Krishna et al., 2017).”
3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera (2019, background): “Examples are on scene graph generation [30, 46], image captioning/description [26, 3, 23], image retrieval [24] and visual question-answering [17, 51].”
Pathologist-level interpretable whole-slide cancer diagnosis with deep learning (2019, methodology): “To be specific, given an encoded image representation in a 2,048-dimensional vector (by averaging V along the 6 × 6 spatial dimension), c-LSTM takes the vector as the input x 0c at the first time step and begins to generate concept states, ).”
Chunk-based Decoder for Neural Machine Translation (2017, methodology): “Hierarchical RNN s are used for various NLP tasks such as machine translation (Luong and Manning, 2016), document modeling (Li et al., 2015; Lin et al., 2015), dialog generation (Serban et al., 2017), image captioning (Krause et al., 2016), and video captioning (Yu et al., 2016).”