Towards Automatic Learning of Procedures From Web Instructional Videos

By Luowei Zhou, Chenliang Xu, Jason J. Corso. 2017

1,070 citations recorded by the source corpus.

The potential for agents, whether embodied or software, to learn by observing other agents performing procedures involving objects and actions is rich. Current research on automatic procedure learning heavily relies on action labels or video subtitles, even during the evaluation phase, which makes them infeasible in real-world scenarios. This leads to our question: can the human-consensus structure of a procedure be learned from a large set of long, unconstrained videos (e.g., instructional videos from YouTube) with only visual evidence? To answer this question, we introduce the problem of procedure segmentation---to segment a video procedure into category-independent procedure segments. Given that no large-scale dataset is available for this problem, we collect a large-scale procedure segmentation dataset with procedure segments temporally localized and described; we use cooking videos and name the dataset YouCook2. We propose a segment-level recurrent network for generating procedure segments by modeling the dependencies across segments. The generated segments can be used as pre-processing for other tasks, such as dense video captioning and event parsing. We show in our experiments that the proposed model outperforms competitive baselines in procedure segmentation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

VideoBERT: A Joint Model for Video and Language Representation Learning (2019, methodology): “6, we show that our approach to video captioning significantly outperforms the previous state-of-the-art [39] on the YouCook II dataset [38].”
Learning Video Representations from Textual Web Supervision (2020, background): “Prior works have leveraged ambient sound [52, 5, 4, 71, 34, 51, 55], dialogue [49], and narration [70, 2, 72, 73, 46, 45, 3], all of which of which serve as useful signals.”
Self-Supervised MultiModal Versatile Networks (2020, methodology): “We are also not far from the stateof-the-art performance reported in [52] for MSRVTT (32.2) and still below for YouCook2 (51.2).”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, background): “Text-to-Video retrieval: YouCook2 [90], MSR-VTT [87].”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “Numerous datasets of web instructional videos were proposed over the past years [2, 30, 45, 47, 50, 67, 68].”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “Table 6 shows the performance of the representative approaches used in Table 2 on TVQA, How2QA, TVC, and YC2C.”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “6-9 on MSRVTT [1], DiDeMo [79], LSMDC [3], and YouCook2 [13], respectively.”
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021, methodology): “Youcook2 (Zhou et al., 2017) contains 2,000 cooking videos on 89 recipes with 14K video clips from YouTube.”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, background): “However, their development mostly proceeds on two independent tracks: one for images , with its own architectures, training datasets and benchmarks [30, 36, 59]; and the other for videos with a similar separation of training datasets and benchmarks [3, 5, 29, 53, 72, 78].”
Unifying Vision-and-Language Tasks via Text Generation (2021, background): “, 2018b) in a wide range of discriminative (Goyal et al., 2019; Hudson & Manning, 2019; Lei et al., 2018; Mao et al., 2016; Xu et al., 2016; Zhou et al., 2018) and generative tasks (Chen et al.”
ActBERT: Learning Global-Local Video-Text Representations (2020, methodology): “We evaluate the ﬁve tasks on CrossTask [59], COIN [37], YouCook2 [53], MSR-VTT [46] and LSMDC [32].”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “, 2020) are the only existing work exploring this space, covering downstream tasks from textbased video retrieval (Zhou et al., 2018a; Xu et al., 2016b) and video question answering (Maharaj et al.”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, methodology): “Finally, we evaluate our model on both In-domain Youcook2 (Zhou et al., 2018a) dataset and Out-domain MSR-VTT (Xu et al.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “, 2016), YouCook2 (Zhou et al., 2018) (results in supplementary materials.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “In instructional video retrieval datasets [29,99], the corresponding YouTube subtitle is only considered relevant, again ignoring semantic overlap or action similarities.”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, neutral): “Commonsense Reasoning Our task unifies work involving reasoning about commonsense phenomena, such as physics [54, 84], social interactions [2, 77, 12, 27], procedure understanding [91, 3] and predicting what might happen next in a video [74, 17, 92, 78, 18, 64, 85].”
Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, methodology): “The YouCook2 dataset [39] contains 2000 Youtube videos of an average length of 5.”
End-to-End Dense Video Captioning with Masked Transformer (2018, positive): “More recently, inspired by the anchoring mechanism from object detection [25], two types of methods have been proposed— explicit anchoring [11, 42] and implicit anchoring [8, 4].”