Unsupervised Semantic Parsing of Video Collections

By Ozan Sener, Amir Zamir, Silvio Savarese, Ashutosh Saxena. 2015

107 citations recorded by the source corpus.

Human communication typically has an underlying structure. This is reflected in the fact that in many user generated videos, a starting point, ending, and certain objective steps between these two can be identified. In this paper, we propose a method for parsing a video into such semantic steps in an unsupervised way. The proposed method is capable of providing a semantic "storyline" of the video composed of its objective steps. We accomplish this utilizing both visual and language cues in a joint generative model. The proposed method can also provide a textual description for each of identified semantic steps and video segments. We evaluate this method on a large number of complex YouTube videos and show results of unprecedented quality for this new and impactful problem.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Self-Supervised MultiModal Versatile Networks (2020, background): “Recently, instructional videos became a popular source of video and language data [6, 51, 75, 94] due to not requiring careful manual annotation, e.”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “[47] use WikiHow, an encyclopedia of how to articles, to collect 17 popular physical tasks, and obtain videos by querying these tasks on YouTube.”
Joint Discovery of Object States and Manipulation Actions (2017, background): “in instructional videos [3, 32, 37] or while performing daily activities [38].”
Unsupervised Learning from Narrated Instruction Videos (2015, background): “The recent concurrent work [27] is addressing, independently of our work, a similar problem but with a different approach based on a probabilistic generative model and considering a different set of tasks mainly focussed on cooking activities.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Prior works have used web videos with ASR to build weakly-supervised object detectors [87], action detectors/classiﬁers [120, 6, 62, 84], instruction aligners [77, 5, 19], video captioners [96, 46, 86, 101], and visual reference resolvers [49].”
Towards Automatic Learning of Procedures From Web Instructional Videos (2017, methodology): “Existing datasets for analyzing instructional videos suffer from either limited videos (Alayrac et al. 2016; Sener et al. 2015) or weak diversity in background (Regneri et al. 2013) and activities (Kuehne, Arslan, and Serre 2014)).”
Localizing Moments in Video with Natural Language (2017, background): “In a similar vein, [1, 4, 33] consider aligning textual instructions to videos.”
Cross-Task Weakly Supervised Learning From Instructional Videos (2019, methodology): “, like [1, 18, 22, 29]) and jointly learn when steps occur and what they look like.”
COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis (2019, methodology): “With the explosion of video data on the Internet, people around the world have uploaded and watched substantial instructional videos [10, 36], covering miscellaneous categories.”
Connectionist Temporal Modeling for Weakly Supervised Action Labeling (2016, background): “Our work is also related to recent progress on using instructional videos or movie scripts [2,4,9,25,30,35,51] as supervision for video parsing.”
VirtualHome: Simulating Household Activities Via Programs (2018, background): “[20, 3] also argued for actions as a sequence of atomic steps.”
A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions (2019, background): “formative in a variety of instructional video understanding tasks (Naim et al., 2014, 2015; Malmaud et al., 2015; Sener et al., 2015; Alayrac et al., 2016; Huang et al., 2017); though less work has focused on instructional caption generation, which”
Temporal Cycle-Consistency Learning (2019, background): “Past work has also explored parsing of fine-grained actions in videos [24, 25, 29] while some others [13, 33, 34, 36] discover sub-activities without explicit supervision of temporal boundaries.”
RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes (2018, background): “…include parsing of recipes (Mal-maud et al., 2014; Jermsurawong and Habash, 2015), aligning instructional text to videos (Mal-maud et al., 2015; Sener et al., 2015), recipe text generation (Kiddon et al., 2016), learning cross-modal embeddings (Salvador et al., 2017), tracking entities and…”
D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation (2019, background): “Action recognition has been an important task for video understanding [13, 27, 33, 36].”
Dense Procedure Captioning in Narrated Instructional Videos (2019, methodology): “Graph-based clustering is also adopted to learn the semantic storyline of instructional videos in (Sener et al., 2015).”