Coherent Multi-sentence Video Description with Variable Level of Detail
By Anna Rohrbach, Marcus Rohrbach, Weijian Qiu, Annemarie Friedrich, Manfred Pinkal, Bernt Schiele. 2014
230 citations recorded by the source corpus.
Humans can easily describe what they see in a coherent way and at varying level of detail. However, existing approaches for automatic video description focus on generating only single sentences and are not able to vary the descriptions’ level of detail. In this paper, we address both of these limitations: for a variable level of detail we produce coherent multi-sentence descriptions of complex videos. To understand the difference between detailed and short descriptions, we collect and analyze a video description corpus of three levels of detail. We follow a two-step approach where we first learn to predict a semantic representation (SR) from video and then generate natural language descriptions from it. For our multi-sentence descriptions we model across-sentence consistency at the level of the SR by enforcing a consistent topic. Human judges rate our descriptions as more readable, correct, and relevant than related work.
Subfields: Computer Vision and Pattern Recognition, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Trevor Darrell
- Dacheng Tao
- Xiaogang Wang
- Yi Yang
- Andrew Zisserman
- Jitendra Malik
- Ming-Hsuan Yang
In Natural Language Generation
- Kate Saenko
- Marcus Rohrbach
- Lisa Anne Hendricks
- Subhashini Venugopalan
- Trevor Darrell
- Jeff Donahue
- Peter Anderson
- Anna Rohrbach
Citations in the corpus
- VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, background): “Various datasets for video description/captioning have been introduced to empower different ways to describe the video content, covering a wide range of domains, such as cooking [16, 72, 45, 46], movie [56, 47, 48], social media [22], and human activities [14, 52, 66, 30].”
- Long-term recurrent convolutional networks for visual recognition and description (2014, background): “8 Table 5: Video description: Results on detailed description of TACoS multilevel[29], in %, see Section C.”
- Dense-Captioning Events in Videos (2017, methodology): “Orthogonally, in an effort to increase the complexity of descriptions, TACos multilevel [12] expanded the TACoS [11] dataset to include paragraph descriptions to instructional cooking videos.”
- A dataset for Movie Description (2015, methodology): “For them we train the visual classifiers as in [58].”
- Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding (2016, background): “Standard approaches in the past have used videos downloaded from the Internet [3,4,5,6,7,8] gathered from movies [9,10,11] or recorded in controlled environments [12,13,14,15,16,17].”
- Localizing Moments in Video with Natural Language (2017, background): “TACoS multi-level[30] 185/14,105 52,593 Lab Kitchen Cooking X X”
- Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks (2015, methodology): “We compare our approach on TACoS-MultiLevel with three state-of-the-art methods: CRF-T [37], CRF-M [36], and LRCN [11].”
- Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research (2015, background): “The majority of available video datasets are toy domains with with small word vo-cabularies such as TaCos [17], TaCos Multi-Level [18], YouCook [11], and MSVD [9].”
- Towards a Visual Turing Challenge (2014, methodology): “tanding [9, 10, 11, 12, 13] tasks. The progress in both fields has inspired researchers to build holistic architectures for challenging grounding [14, 15], natural language generation from image/video [16, 17, 18], image-to-sentence alignment [19, 20, 21, 22], and recently presented question-answerin”
- Benchmarking Hierarchical Script Knowledge (2019, background): “Rohrbach et al. (2014) describe a similar corpus of natural descriptions of composite actions, useful for activity recognition in video.”
- Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning (2019, methodology): “More recent methods [50, 39] rely on Deep Learning [28] to build frameworks resembling a typical neural machine translation system that can generate a single sentence [57, 33] or multiple sentences [38, 43, 60] to describe videos.”
- Reconstruction Network for Video Captioning (2018, methodology): “In this section, we first introduce two types of video cap-tioning: template-based approaches [17, 10, 30, 29, 48] and sequence learning approaches [49, 39, 40, 8, 27, 14, 52, 24, 25, 32, 19], then introduce the application of dual learning.”
- Recurrent Topic-Transition GAN for Visual Paragraph Generation (2017, methodology): “As in Table 4, we also extend our RTT-GAN to the task of video paragraph generation and evaluate it on TACoSMultiLevel dataset [24] that contains 185 long videos filmed in an indoor environment, following [35].”
- Physical Causality of Action Verbs in Grounded Language Understanding (2016, methodology): “To acquire knowledge of verb causality, we collected a larger dataset of causality annotations based on sentences from the TACoS Multilevel corpus (Rohrbach et al., 2014), through crowd-sourcing on Amazon Mechanical Turk.”