Unsupervised Learning from Narrated Instruction Videos

By Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, Simon Lacoste-Julien. 2015

323 citations recorded by the source corpus.

We address the problem of automatically learning the main steps to complete a certain task, such as changing a car tire, from a set of narrated instruction videos. The contributions of this paper are three-fold. First, we develop a new unsupervised learning approach that takes advantage of the complementary nature of the input video and the associated narration. The method solves two clustering problems, one in text and one in video, applied one after each other and linked by joint constraints to obtain a single coherent sequence of steps in both modalities. Second, we collect and annotate a new challenging dataset of real-world instruction videos from the Internet. The dataset contains about 800,000 frames for five different tasks1 that include complex interactions between people and objects, and are captured in a variety of indoor and outdoor settings. Third, we experimentally demonstrate that the proposed method can automatically discover, in an unsupervised manner, the main steps to achieve the task and locate the steps in the input videos.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

VideoBERT: A Joint Model for Video and Language Representation Learning (2019, background): “, [16, 2, 10, 38, 39]) have trained models to analyse instructional videos, such as cooking.”
Learning Video Representations from Textual Web Supervision (2020, background): “Prior works have leveraged ambient sound [52, 5, 4, 71, 34, 51, 55], dialogue [49], and narration [70, 2, 72, 73, 46, 45, 3], all of which of which serve as useful signals.”
Self-Supervised MultiModal Versatile Networks (2020, background): “Recently, instructional videos became a popular source of video and language data [6, 51, 75, 94] due to not requiring careful manual annotation, e.”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, positive): “To avoid labeling visual data, several approaches have leveraged audio transcripts obtained from narrated videos using automatic speech recognition (ASR) as a way to supervise video models for object detection [4, 16, 56], captioning [34, 72], classification [3, 44, 49, 89], summarization [59] or re”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “Related to our work, [2, 30, 62] also consider automatically generated transcription of narrated instructional videos as a source of supervision.”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “Please see the supplemental material for example demonstrations and language for each task. and concepts [1, 45, 42, 55, 58].”
Affine Invariant Analysis of Frank-Wolfe on Strongly Convex Sets (2020, background): “They form a practical family of algorithms [Jaggi 2013; Bojanowski et al. 2014; Alayrac et al. 2016; Seguin et al. 2016; Peyre et al. 2017; Miech x k +1 = (1 − γ t ) x k + γ k v k ⊲ Convex update 5: end for et al. 2018; Lacoste-Julien et al. 2015b; Courty et al. 2016; Paty et al. 2019; Luise et al.…”
Frank-Wolfe Splitting via Augmented Lagrangian Method (2018, background): “Moreover, for some highly structured polytopes such as those appearing in alignment constraints (Alayrac et al., 2016) or Structured SVM (Lacoste-Julien et al.”
Joint Discovery of Object States and Manipulation Actions (2017, methodology): “the approach of [3, 6, 7], each video is divided into chunks of 10 frames that are represented by a motion and appearance descriptor averaged over 30 frames.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Prior works have used web videos+ASR to build weakly-supervised object detectors [75], action detectors/classifiers [108, 5, 51, 72], instruction aligners [65, 4, 15], video captioners [85, 38, 74, 89], and visual reference resolvers [41].”
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021, neutral): “Average Recall Joint Alignment Alayrac (Alayrac et al., 2016) 13.3 Zhukov (Zhukov et al., 2019) 22.4 Supervised (Zhukov et al., 2019) 31.6 HowTo100M (Miech et al., 2019) 33.6 MIL-NCE (Miech et al., 2020) 40.5 UniVL (Luo et al., 2020) 42.0 Pairwise Matching ActBERT (Zhu and Yang, 2020) 41.4 VLM (task”
ActBERT: Learning Global-Local Video-Text Representations (2020, background): “Learning from instructional videos is challenging due to its data complexity across various tasks [6, 1, 51, 26].”
Towards Automatic Learning of Procedures From Web Instructional Videos (2017, methodology): “Existing datasets for analyzing instructional videos suffer from either limited videos (Alayrac et al. 2016; Sener et al. 2015) or weak diversity in background (Regneri et al. 2013) and activities (Kuehne, Arslan, and Serre 2014)).”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, neutral): “Commonsense Reasoning Our task unifies work involving reasoning about commonsense phenomena, such as physics [54, 84], social interactions [2, 77, 12, 27], procedure understanding [91, 3] and predicting what might happen next in a video [74, 17, 92, 78, 18, 64, 85].”
Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, background): “Language can be leveraged by finding a joint embedding space for both visual and textual modalities or by learning an alignment between the two modalities [14, 2, 38].”
Localizing Moments in Video with Natural Language (2017, background): “In a similar vein, [1, 4, 36] consider aligning textual instructions to videos.”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, background): “Towards this goal recent works study representations of vision and language addressing tasks such as visual question answering [1], [2], action learning and discovery [3], [4], [5], text-based event localization [6] as well as video captioning, retrieval and summarization [7], [8], [9], [10].”