Learning from Narrated Instruction Videos

By Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, Simon Lacoste-Julien. 2015

23 citations recorded by the source corpus.

Automatic assistants could guide a person or a robot in performing new tasks, such as changing a car tire or repotting a plant. Creating such assistants, however, is non-trivial and requires understanding of visual and verbal content of a video. Towards this goal, we here address the problem of automatically learning the main steps of a task from a set of narrated instruction videos. We develop a new unsupervised learning approach that takes advantage of the complementary nature of the input video and the associated narration. The method sequentially clusters textual and visual representations of a task, where the two clustering problems are linked by joint constraints to obtain a single coherent sequence of steps in both modalities. To evaluate our method, we collect and annotate a new challenging dataset of real-world instruction videos from the Internet. The dataset contains videos for five different tasks with complex interactions between people and objects, captured in a variety of indoor and outdoor settings. We experimentally demonstrate that the proposed method can automatically discover, learn and localize the main steps of a task in input videos.

Subfields: Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Gradient-Based Neural DAG Learning (2019, background): “Continuous approaches to combinatorial optimization have sometimes demonstrated improved performance over discrete approaches in the literature (see for example Alayrac et al. (2018, § 5.2) where they solve the multiple sequence alignment problem with a continuous optimization method).”
Minding the Gaps for Block Frank-Wolfe Optimization of Structured SVMs (2016, background): “The first three contributions are general to BCFW and thus could be applied to other block-separable optimization problems where BCFW could or have been used such as video colocalization (Joulin et al., 2014), multiple sequence alignment (Alayrac et al., 2016, App.”
Cross-Task Weakly Supervised Learning From Instructional Videos (2019, background): “ategories form two distinct clusters on the diagram. The rest of the tasks form a continuous Cooking cluster. Table2provides some statistics for the primary tasks. The order consistency, deﬁned as in [36], shows how well the order is respected in the videos. For example, if the order of steps in a v”
Connectionist Temporal Modeling for Weakly Supervised Action Labeling (2016, background): “An alternative is to develop methods that require weak supervision, which may be automatically extracted from movie scripts [3,5,9] or instructional videos [2,4,50] at a lower cost.”
Unsupervised Image Matching and Object Discovery as Optimization (2019, background): “Several alternatives are possible, including weakly supervised approaches that rely on readily available meta-data [2, 9] or image-level labels [14, 23, 24, 25, 39, 45] instead of more complex annotations such as bounding boxes [17, 38] or object masks [20] as supervisory signal; semi supervised met”
Dense Procedure Captioning in Narrated Instructional Videos (2019, unknown): “Besides the alignment of transcripts with video, (Alayrac et al., 2016, 2018) propose to learn the main steps from a set of narrated instructional videos for ﬁve different tasks and formulate the problem into two clustering problems.”
A Convex Atomic-Norm Approach to Multiple Sequence Alignment and Motif Discovery (2016, background): “The above objective (1) is also known as the Star Alignment (in distinction to the Sumof-Pairs objective, for which a convex relaxation approach has been considered in (Alayrac et al., 2015)).”
Weakly supervised learning of actions from transcripts (2016, background): “Additionally, transcripts may even be directly extracted from subtitles [23, 24] without any annotation cost at all.”