Integrating Language and Vision to Generate Natural Language Descriptions of Videos in the Wild

By Jesse Thomason, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, Raymond J. Mooney. 2014

201 citations recorded by the source corpus.

Subfields: Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Long-term recurrent convolutional networks for visual recognition and description (2014, methodology): “[11, 30, 17, 3, 6, 17, 41, 42] propose methods for generating sentence descriptions for video, but to our knowledge we present the first application of deep models to the video description task.”
Movie Description (2016, background): “In the past video description has been addressed in controlled settings (Barbu et al. 2012; Kojima et al. 2002), on a small scale (Das et al. 2013; Guadarrama et al. 2013; Thomason et al. 2014) or in single domains like cooking (Rohrbach et al. 2014, 2013; Don-ahue et al. 2015).”
A dataset for Movie Description (2015, methodology): “Our extraction of annotations has similarities to [63], but we try to extract the senses of the words automatically by using semantic parsing as discussed in Section 5.”
Sequence to Sequence -- Video to Text (2015, methodology): “Several previous methods for generating sentence descriptions [11, 19, 36] used a two stage pipeline that first identifies the semantic content (subject, verb, object) and then generates a sentence based on a template.”
Translating Videos to Natural Language Using Deep Recurrent Neural Networks (2014, methodology): “The previous work on the YouTube corpus we employ (Motwani and Mooney, 2012; Krishnamoorthy et al., 2013; Guadarrama et al., 2013; Thomason et al., 2014) used a two-step approach, first detecting a fixed tuple of role words, such as subject, verb, object, and scene, and then using a template to gene”
Describing Videos by Exploiting Temporal Structure (2015, neutral): “47 66.63 + Local + Global 0.4192 0.2960 0.5167 27.55 0.007 0.057 0.061 65.44 Venugopalan et al. [41] 0.3119 0.2687 - - - - - - + Extra Data (Flickr30k, COCO) 0.3329 0.2907 - - - - - - Thomason et al. [37] 0.1368 0.2390 - - - - - - Training For all video description generation models, we estimated th”
Delving Deeper into Convolutional Networks for Learning Video Representations (2015, methodology): “ 0.499 0.326 - Soft-attention Yao et al. (2015b) - 0.4192 0.2960 0.5167 Venugopalan et al. Venugopalan et al. (2015) - 0.3119 0.2687 - + Extra Data (Flickr30k, COCO) - 0.3329 0.2907 - Thomason et al. Thomason et al. (2014) - 0.1368 0.2390 - Table 2: Performance of different variants of the model on ”
Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning (2015, background): “B@1 B@2 B@3 B@4 FGM [33] 23.9 - - - -”
Jointly Modeling Embedding and Translation to Bridge Video and Language (2015, methodology): “We conduct our experiments mainly on the Microsoft Research Video Description Corpus (YouTube2Text) [5], which have been used in several prior works [11, 29, 34] on action recognition and video description generation tasks.”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “…been an increased interest in using visual data as input, resulting in so-called vision-to-text systems which automatically produce descriptions of static or moving images based on computer vision input (e.g., Mitchell et al., 2012; Kulkarni et al., 2013; Thomason et al., 2014, among many others).”
Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data (2015, background): “Many early caption models [31, 17, 8, 18, 9] rely on first discerning visual elements from an image, such as subjects, objects, scenes, and actions, then filling in a sentence template to create a coherent visual description.”
Visual Relationship Detection with Language Priors (2016, background): “Some papers explicitly collected relationships in images [25,26,27,28,29] and videos [27,30,31] and helped models map these relationships from images to language.”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, background): “Advances in simulators and large-scale data collection have facilitated moving past static language grounding tasks like visual question answering [4, 23] and captioning [80, 65, 36] to embodied benchmarks like Vision-and-Language Navigation (VLN) [49, 17, 3, 61, 41], embodied question answering [22”
Long-Term Recurrent Convolutional Networks for Visual Recognition and Description (2017, methodology): “Recent approaches to describing video with natural language have made use of templates, retrieval, or language models [11], [59], [60], [60], [61], [62], [63], [64].”
Motion Guided Spatial Attention for Video Captioning (2019, methodology): “Traditional approaches (Krishnamoorthy et al. 2013; Thomason et al. 2014; Guadarrama et al. 2013) used various visual classiﬁers/trackers to detect visual concepts and then generate sentences with predeﬁned language templates.”
Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning (2019, background): “able 1. Benchmarking on MSVD dataset [11] in terms of BLEU4 (B-4), METEOR (M), CIDEr D (C) and ROUGE L (R). See the text for the description of proposed method GRU-EVE’s variants. Model B-4 M C R FGM [47] 13.7 23.9 - - S2VT [50] - 29.2 - - LSTM-YT [51] 33.3 29.1 - - Temporal-Attention (TA) [59] 41.9”
Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning (2020, background): “Video Captioning: Captioning is crucial for understanding visuals; however it is typically limited to describing observable objects and events (Yang et al., 2011; Thomason et al., 2014; Gan et al., 2017)), or for generating paragraphs or multisentence captions about the image or video (Krause et al.”