Cross-Task Weakly Supervised Learning From Instructional Videos

By Dimitri Zhukov, Jean-Baptiste Alayrac, R. G. Cinbis, David F. Fouhey, Ivan Laptev, Josef Sivic. 2019

314 citations recorded by the source corpus.

In this paper we investigate learning visual models for the steps of ordinary tasks using weak supervision via instructional narrations and an ordered list of steps instead of strong supervision via temporal annotations. At the heart of our approach is the observation that weakly supervised learning may be easier if a model shares components while learning different steps: ``pour egg'' should be trained jointly with other tasks involving ``pour'' and ``egg''. We formalize this in a component model for recognizing steps and a weakly supervised learning framework that can learn this model under temporal constraints from narration and the list of steps. Past data does not permit systematic studying of sharing and so we also gather a new dataset aimed at assessing cross-task sharing. Our experiments demonstrate that sharing across tasks improves performance, especially when done at the component level and that our component model can parse previously unseen tasks by virtue of its compositionality.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, background): “Prior works have leveraged ambient sound [52, 5, 4, 71, 34, 51, 55], dialogue [49], and narration [70, 2, 72, 73, 46, 45, 3], all of which of which serve as useful signals.”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, methodology): “Action Step Localization: CrossTask [91].”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “In a similar vein, COIN [50] and CrossTask [68] datasets are collected by first searching for tasks on WikiHow and then videos for each task on YouTube.”
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021, methodology): “CrossTask (Zhukov et al., 2019) is a dataset for action localization that contains 83 different tasks and 4.7k videos.”
ActBERT: Learning Global-Local Video-Text Representations (2020, methodology): “CrossTask [59] contains 83 tasks and 4.7k videos related to cooking, car maintenance, crafting, etc.”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, methodology): “We fine-tune and evaluate our model on two instructional video datasets: YouCook2 [23] and CrossTask [30].”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, methodology): “Lastly, we report VideoCLIP’s performance on CrossTask in Table 5.”
Use What You Have: Video retrieval using representations from collaborative experts (2019, background): “More recently, alternative approaches to addressing the issue of annotation scarcity have been explored, which include self-supervised [47] and weakly-supervised [57] video-text models.”
Look Before you Speak: Visually Contextualized Utterances (2020, methodology): “CrossTask-NSP: CrossTask [90] is another dataset that contains manual annotations of steps for instructional videos of 18 pre-selected tasks.”
Understanding Human Hands in Contact at Internet Scale (2020, background): “Of the many works in video human-object interaction [11, 39, 33, 24, 40], ours is most clearly related to VLOG [14], which also gathers data of interaction, and AVA [18], which investigates atomic (i.e., base-level) actions.”
TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment (2021, methodology): “Using the official algorithm provided by [60], we can find the optimal framewise order of action steps for a video.”
Procedure Planning in Instructional Videos (2019, methodology): “We use the CrossTask instructional video dataset [27] collected for 83 different tasks for our evaluations.”
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, unknown): “[10, 126, 145, 148] or third-person [9, 100, 159, 205, 207] videos; however, limited work considers both views together.”
Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities (2022, background): “Multi-step activities are best exemplified in cooking and instructional videos, so the majority of datasets in this area are curated from online video platforms, e.g. YouTube Instructional [1], What’s Cooking [27], YoucookII [50], CrossTask [52], COIN [49] and HowTo100M [29].”
PlaTe: Visually-Grounded Planning With Transformers in Procedural Tasks (2021, methodology): “Datasets: We first evaluate PlaTe on an instructional video dataset CrossTask [16].”
Action Modifiers: Learning From Adverbs in Instructional Videos (2019, methodology): “However, in [62], only 30 videos per task were used in training.”