YouTube-8M: A Large-Scale Video Classification Benchmark

By Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Apostol Natsev, George Toderici, Balakrishnan Varadarajan, Sudheendra Vijayanarasimhan. 2016

1,406 citations recorded by the source corpus.

Many recent advancements in Computer Vision are attributed to large datasets. Open-source software packages for Machine Learning and inexpensive commodity hardware have reduced the barrier of entry for exploring novel approaches at scale. It is possible to train models over millions of examples within a few days. Although large-scale datasets exist for image understanding, such as ImageNet, there are no comparable size video classification datasets. In this paper, we introduce YouTube-8M, the largest multi-label video classification dataset, composed of ~8 million videos (500K hours of video), annotated with a vocabulary of 4800 visual entities. To get the videos and their labels, we used a YouTube video annotation system, which labels videos with their main topics. While the labels are machine-generated, they have high-precision and are derived from a variety of human-based signals including metadata and query click signals. We filtered the video labels (Knowledge Graph entities) using both automated and manual curation strategies, including asking human raters if the labels are visually recognizable. Then, we decoded each video at one-frame-per-second, and used a Deep CNN pre-trained on ImageNet to extract the hidden representation immediately prior to the classification layer. Finally, we compressed the frame features and make both the features and video-level labels available for download. We trained various (modest) classification models on the dataset, evaluated them using popular evaluation metrics, and report them as baselines. Despite the size of the dataset, some of our models train to convergence in less than a day on a single machine using TensorFlow. We plan to release code for training a TensorFlow model and for computing metrics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “Specifically, three NoisyStudent [70] variants which use IMAGENET and unlabelled data from the JFT dataset, BiT (BigTransfer) [36] models that have been first trained on IMAGENET, IMAGENET-21K, or JFT and then transferred to IMAGENET by fine-tuning, and the Video-Induced Visual Invariance (VIVI) mod”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “As a result, and with the recent emergence of research video data sets [1, 59], videos have been investigated in the context of self-supervision (for example, [40, 64, 63, 28, 65, 73, 41, 51, 42, 3, 2]).”
Learning Video Representations from Textual Web Supervision (2020, methodology): “Other approaches use user-deﬁned keywords or tags [20, 28, 43, 19] or algorithmically-generated topics [1, 31] to the same end.”
Self-Supervised MultiModal Versatile Networks (2020, background): “Dataset abbreviations: AudioSet, HowTo100M, Instagram65M [26], SoundNet [10], 2M videos from YouTube8M [5], Kinetics600; their length in years is given in the “years” column.”
Learning from Simulated and Unsupervised Images through Adversarial Training (2016, background): “Large labeled training datasets are becoming increasingly important with the recent rise in high capacity deep neural networks [4, 20, 48, 48, 1, 24, 17].”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “[1] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and SudheendraVijayanarasimhan.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Finally, we do not plan on releasing videos for download, only their IDs, following a strategy from prior work [1, 80].”
AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions (2017, background): “Recently, datasets, such as TrecVid MED [29], Sports-1M [21], YouTube-8M [1], Something-something [12], SLAC [48], Moments in Time [28], and Kinetics [22] have focused on large-scale video classiﬁcation, often with automatically generated – and hence potentially noisy – annotations.”
UvA-DARE (Digital Academic Repository) Self-Supervised Video Representation Learning With Odd-One-Out Networks (neutral): “Second, existing large video datasets, e.g. , Sports-1M [23] and the recent YouTube-8M [1], rely on noisy, un-reliable YouTube tags.”
Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition (2017, background): “More recently, huge databases such as Sports-1M [11] and YouTube-8M [1] are proposed.”
FearNet: Brain-Inspired Model for Incremental Learning (2017, methodology): “For AudioSet, we use the audio CNN embeddings produced by pre-training the model on the YouTube-8M dataset (Abu-El-Haija et al., 2016).”
Measuring Catastrophic Forgetting in Neural Networks (2017, methodology): “2017), which was pre-trained on an early version of the YouTube-8m dataset (Abu-El-Haija et al. 2016).”
CORe50: a New Dataset and Benchmark for Continuous Object Recognition (2017, background): “YouTube-8M [1] provides a huge number of videos acquired in difficult natural settings.”
Pose-conditioned Spatio-Temporal Attention for Human Action Recognition (2017, background): “Performance close to stateof-the-art is achieved on the smaller MSR Daily Activity 3D dataset.”
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset (2018, background): “This has been changing recently, with the release of the action classification benchmarks such as [18, 1, 54, 38, 46, 14].”
Video Action Transformer Network (2018, background): “Datasets have become progressively larger and harder: from actors performing simple actions [12, 33], to short sports and movie clips [24, 38], finally to diverse youtube videos [1, 23].”