Understanding videos, constructing plots learning a visually grounded storyline model from annotated videos

By Abhinav Kumar Gupta, Praveen Srinivasan, Jianbo Shi, Larry S. Davis. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009

299 citations recorded by the source corpus.

Analyzing videos of human activities involves not only recognizing actions (typically based on their appearances), but also determining the story/plot of the video. The storyline of a video describes causal relationships between actions. Beyond recognition of individual actions, discovering causal relationships helps to better understand the semantic meaning of the activities. We present an approach to learn a visually grounded storyline model of videos directly from weakly labeled data. The storyline model is represented as an AND-OR graph, a structure that can compactly encode storyline variation across videos. The edges in the AND-OR graph correspond to causal relationships which are represented in terms of spatio-temporal constraints. We formulate an Integer Programming framework for action recognition and storyline extraction using the storyline model and visual groundings learned from training data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/49927656ede0c75af22ca73dcf4abba028839650

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration (2018, unknown): “This includes instructional video [2, 18, 34], movie script [37, 45], and caption annotation [14, 25].”
A dataset for Movie Description (2015, background): “In recent years there has been an increased interest in automatically describing images [23, 39, 40, 50, 45, 40, 41, 34, 61, 22] and videos [37, 27, 8, 28, 32, 62, 16, 26, 64, 55] with natural language.”
Finding Actors and Actions in Movies (2013, methodology): “In video, manually provided text descriptions have been used to learn a causal model of human actions in the constrained domain of sports events [14].”
Baby talk: Understanding and generating simple image descriptions (2011, methodology): “In addition to these generation efforts in images, there has also been work related to linking humans and their actions in video [21], [23], [24].”
End-to-End Learning of Action Detection from Frame Glimpses in Videos (2015, unknown): “2 shows our model structure.”
A survey on activity recognition and behavior understanding in video surveillance (2013, background): “In [55], work on activity classification has focused on recognition of atomic level action, using 2D video.”
Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning (2020, background): “The above example also illustrates that recognition of objects, actions, and events is often not enough; understanding causal relationships, social interactions, and commonsense aspects behind them provides context and a more semantic interpretation of the video (Gupta et al., 2009).”
Image Retrieval with Structured Object Queries Using Latent Ranking SVM (2012, methodology): “[3] use the AND-OR graph formalism to represent spatiotemporal relations among objects and actions in videos.”
Beyond Actions: Discriminative Models for Contextual Group Activities (2010, unknown): “[16]’s representation based on AND-OR graphs allows for a flexible grammar of action relationships.”
Video-Based Human Behavior Understanding: A Survey (2013, methodology): “[109] used a storyline to describe causal relationships between actions (e.”
Patch to the Future: Unsupervised Visual Prediction (2014, methodology): “For longer term predictions, models such as Markov Decision Process (MDP) [18, 35], Markov Logic Networks [29], ATCRF [19], CRF [8], and AND-OR graphs [12, 25] are used.”
Visual Semantic Role Labeling (2015, methodology): “While these methods signiﬁcantly boost performance over not using human object context and produce localization for the object of interaction as learned by their model, they do not quantify performance at the joint task of detecting people, classifying what they are doing and localizing the object…”
Translating Video Content to Natural Language Descriptions (2013, methodology): “[9] extracts an AND-OR graph from sports videos to model causal relationships.”
Virtual Embodiment: A Scalable Long-Term Strategy for Artificial Intelligence Research (2016, background): “th the environment, and build more intricate “reﬂective reasoning” on top of that foundation [19]. There has been work in linguistic grounding that allows for temporal aspects, for instance in videos [20, 21, 22], and both time and interaction, notably in the ﬁe ld of robotics [23, 24, 25]. However,”
Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos (2017, background): “This includes recent progress in scene graphs [13, 20, 51, 64], storylines [1, 14, 17, 31, 53], and action understanding [10, 44, 55].”