Generating Natural-Language Video Descriptions Using Text-Mined Knowledge

By Niveda Krishnamoorthy, Girish Malkarnenkar, Raymond J. Mooney, Kate Saenko, Sergio Guadarrama. 2013

246 citations recorded by the source corpus.

We present a holistic data-driven technique that generates natural-language descriptions for videos. We combine the output of state-of-the-art object and activity detectors with "real-world' knowledge to select the most probable subject-verb-object triplet for describing a video. We show that this knowledge, automatically mined from web-scale text corpora, enhances the triplet selection algorithm by providing it contextual information and leads to a four-fold increase in activity identification. Unlike previous methods, our approach can annotate arbitrary videos without requiring the expensive collection and annotation of a similar training video corpus. We evaluate our technique against a baseline that does not use text-mined knowledge and show that humans prefer our descriptions 61% of the time.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks (2015, methodology): “To eliminate the tedious effort of rule engineering when the problem scales, some recent methods train statistical models for lexical entries, either in a fully [10, 14, 24, 41] or weakly [36, 35, 57, 54] supervised fashion.”
Jointly Modeling Deep Video and Compositional Text to Bridge Vision and Language in a Unified Framework (2015, background): “2009; Krishnamoorthy et al. 2013; Guadarrama et al. 2013; Thomason et al. 2014), as depicted in Fig.”
Coherent Multi-sentence Video Description with Variable Level of Detail (2014, methodology): “To generate descriptions for videos and images, rules or templates are a powerful tool but need to be manually de-ﬁned [12, 22, 6, 11, 5].”
Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data (2015, background): “Many early caption models [31, 17, 8, 18, 9] rely on first discerning visual elements from an image, such as subjects, objects, scenes, and actions, then filling in a sentence template to create a coherent visual description.”
See No Evil, Say No Evil: Description Generation from Densely Labeled Images (2014, background): “These problems also extend to video caption systems (Yu and Siskind, 2013; Krishnamoorthy et al., 2013).”
Deep correlation for matching images and text (2015, background): “In parallel to image captioning, automatic video description is also receiving increasing attention [6, 15, 23, 42, 43, 47].”
Integrating Language and Vision to Generate Natural Language Descriptions of Videos in the Wild (2014, methodology): “• Our model includes scene (location) information which has not been addressed by previous video description works (Barbu et al., 2012; Krishnamoorthy et al., 2013; Guadarrama et al., 2013).”
Visual Madlibs: Fill in the blank Image Generation and Question Answering (2015, background): “The ﬁrst type of approach focused on detecting content elements such as objects, attributes, activities, or spatial relationships and then composing captions for images [18, 33, 26, 10] or videos [17] using linguistically inspired templates.”
Write a Classifier: Zero-Shot Learning Using Purely Textual Descriptions (2013, background): “…c(t∗), such that it can be directly used to classify any test image x as c(t∗)T · x > 0 if x belongs to C c(t∗)T · x 0 otherwise (1) In what follows, we introduce two possible frameworks for this problem and discuss potential limitations for them, which leads next to the proposed formulation.”
Title Generation for User Generated Videos (2016, background): “Early work on video captioning [7,9,10,11,12,13,14] typically perform a two-stage procedure.”
Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning (2019, background): “[27] led the early works of describing open domain videos.”
MovieQA: Understanding Stories in Movies through Question-Answering (2015, methodology): “In [10, 41], the authors caption video clips using LSTMs, [33] formulates description as a machine translation model, while older work uses templates [3, 8, 18].”
A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching (2013, unknown): “, and generate language description by template filling; [19] additionally uses externally mined language data to help rank the best subject-verbobject triplet.”
Actionness Ranking with Lattice Conditional Ordinal Random Fields (2014, methodology): “These methods are enabling futuristic vision applications like automatic video-to-text [5, 18] and smart Time”
Semantic Aware Video Transcription Using Random Forest Classifiers (2014, methodology): “Finally, even for the detectors with corresponding SVO terms, many of them are still far from reliable when applied to videos in the wild [15].”
Sequence to Sequence -- Video to Text (2015, methodology): “Several previous methods for generating sentence descriptions [11, 19, 36] used a two stage pipeline that first identifies the semantic content (subject, verb, object) and then generates a sentence based on a template.”
Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text (2016, methodology): “Many of the earlier methods (Yu and Siskind, 2013; Rohrbach et al., 2013; Krishnamoorthy et al., 2013; Guadarrama et al., 2013; Thomason et al., 2014) followed a two-step approach where they first identify the semantic content (e.”