Video In Sentences Out

By Andrei Barbu, Alexander Bridge, Zachary Burchill, D. Coroian, Sven J. Dickinson, Sanja Fidler, Aaron Michaux, Sam Mussman, Siddharth Narayanaswamy, D. Salvi and 8 more. 2012

157 citations recorded by the source corpus.

We present a system that produces sentential descriptions of video: who did what to whom, and where and how they did it. Action class is rendered as a verb, participant objects as noun phrases, properties of those objects as adjectival modifiers in those noun phrases,spatial relations between those participants as prepositional phrases, and characteristics of the event as prepositional-phrase adjuncts and adverbial modifiers. Extracting the information needed to render these linguistic entities requires an approach to event recognition that recovers object tracks, the track-to-role assignments, and changing body posture.

Subfields: Computer Vision and Pattern Recognition, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Generation

Citations in the corpus

Long-term recurrent convolutional networks for visual recognition and description (2014, methodology): “[11, 30, 17, 3, 6, 17, 41, 42] propose methods for generating sentence descriptions for video, but to our knowledge we present the first application of deep models to the video description task.”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “Several approaches generate image captions based on fixed templates that are filled based on the content of the image [19, 29, 13, 55, 56, 9, 1] or generative grammars [42, 57], but this approach limits the variety of possible outputs.”
MovieQA: Understanding Stories in Movies through Question-Answering (2015, methodology): “In [10, 41], the authors caption video clips using LSTMs, [33] formulates description as a machine translation model, while older work uses templates [3, 8, 18].”
A dataset for Movie Description (2015, background): “In recent years there has been an increased interest in automatically describing images [23, 39, 40, 50, 45, 40, 41, 34, 61, 22] and videos [37, 27, 8, 28, 32, 62, 16, 26, 64, 55] with natural language.”
Translating Videos to Natural Language Using Deep Recurrent Neural Networks (2014, background): “Solutions have been proposed for narrow domains with a small set of known actions and objects, e.g., (Barbu et al., 2012; Rohrbach et al., 2013), but generating descriptions for “in-the-wild” videos such as the YouTube domain (Figure 1) remains an open challenge.”
Describing Videos by Exploiting Temporal Structure (2015, background): “Video description generation has been investigated and studied in other work, such as [19, 2, 26].”
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks (2015, methodology): “Following their work, several succeeding approaches [26, 20, 21, 15, 3] applied similar rule-based systems to datasets with larger numbers of objects and events, in different tasks and scenarios.”
Jointly Modeling Deep Video and Compositional Text to Bridge Vision and Language in a Unified Framework (2015, background): “…(Song, Yang, and Huang 2011), language retrieval (Das, Srihari, and Corso 2013) and ultimately, natural language generation (NLG) from novel videos (Barbu et al. 2012; Das et al. 2013; Rohrbach et al. 2013; Gupta et al. Copyright c 2015, Association for the Advancement of Artificial Intelligence…”
Unsupervised Semantic Parsing of Video Collections (2015, background): “Recently, event recounting has attracted much interest and intends to identify the evidential segments for which a video belongs to a certain class [57, 9, 3].”
Automatic Construction and Natural-Language Description of Nonparametric Regression Models (2014, background): “However, systems with natural language output have been built in the areas of video interpretation (Barbu et al., 2012) and automated theorem proving (Ganesalingam and Gowers, 2013).”
Multimodal Machine Learning: A Survey and Taxonomy (2017, background): “[15] proposed a video description model that generates sentences of the form: who did what to whom and where and how they did it.”
Semantic Aware Video Transcription Using Random Forest Classifiers (2014, methodology): “[1] used object detection and tracking results to describe videos with simple actions and fixed camera.”
What Are You Talking About? Text-to-Image Coreference (2014, background): “Most recent approaches that employ text and images focus on generation tasks, where given an image one is interested in generating a lingual description of the scene [8, 12, 21, 2], or given a sentence, retrieving related images [29].”
Long-Term Recurrent Convolutional Networks for Visual Recognition and Description (2017, methodology): “Recent approaches to describing video with natural language have made use of templates, retrieval, or language models [11], [59], [60], [60], [61], [62], [63], [64].”
Translating Video Content to Natural Language Descriptions (2013, methodology): “[10, 2] extract actions, body-pose, objects and their tracks on the DARPA Mind’s eye corpus which depict 48 different verbs.”
Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning (2019, methodology): “osed a method to automatically describe events involving multiple actions (seven on average), performed by one or more individuals. Whilst most of the prior work was restricted to constrained domains [25, 9], Krishnamoorthy et al. [27] led the early works of describing open domain videos. [20] propo”
Trainable performance upper bounds for image and video captioning (2015, background): “xtra semantic information to guide the language generation process. 2.2 VIDEO CAPTIONING Similarly, video captioning has made substantial progress recently. Early models such as Kojima et al. (2002); Barbu et al. (2012); Rohrbach et al. (2013) tend to focus on constrained domains with limited appear”