Mining YouTube - A dataset for learning fine-grained action concepts from webly supervised video data

By Hilde Kuehne, Ahsan Iqbal, Alexander Richard, Juergen Gall. 2019

18 citations recorded by the source corpus.

Action recognition is so far mainly focusing on the problem of classification of hand selected preclipped actions and reaching impressive results in this field. But with the performance even ceiling on current datasets, it also appears that the next steps in the field will have to go beyond this fully supervised classification. One way to overcome those problems is to move towards less restricted scenarios. In this context we present a large-scale real-world dataset designed to evaluate learning techniques for human action recognition beyond hand-crafted datasets. To this end we put the process of collecting data on its feet again and start with the annotation of a test set of 250 cooking videos. The training data is then gathered by searching for the respective annotated classes within the subtitles of freely available videos. The uniqueness of the dataset is attributed to the fact that the whole process of collecting the data and training does not involve any human intervention. To address the problem of semantic inconsistencies that arise with this kind of training data, we further propose a semantical hierarchical structure for the mined classes.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, methodology): “The most commonly-used approach is to use image search results, and to label each image with the query that was used to ﬁnd it [65, 15, 56, 6, 11, 13, 21, 10, 18, 35].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, methodology): “To avoid labeling visual data, several approaches have leveraged audio transcripts obtained from narrated videos using automatic speech recognition (ASR) as a way to supervise video models for object detection [4, 16, 56], captioning [34, 72], classification [3, 44, 49, 89], summarization [59] or re”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Prior works have used web videos with ASR to build weakly-supervised object detectors [87], action detectors/classiﬁers [120, 6, 62, 84], instruction aligners [77, 5, 19], video captioners [96, 46, 86, 101], and visual reference resolvers [49].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, neutral): “The recent inﬂux of instructional video datasets such as How2 [28], Inria Instructional Videos [60], COIN [61], CrossTask [31], YouCook2 [23], Mining YouTube [62], and HowTo100M [25] has inspired a variety of methods for semi-supervised text-video modelling.”
Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube (2020, unknown): “…et al., 2017; Huang* et al., 2018; Moriya et al., 2019), e.g., action detection/classification (Yu et al., 2014; Alayrac et al., 2017; Chang et al., 2019; Kuehne et al., 2019), segmentation/captioning (Sener et al., 2015), and instruction alignment (Malmaud et al., 2015; Alayrac et al., 2016).”