The Kinetics Human Action Video Dataset

By Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Apostol Natsev and 2 more. 2017

4,530 citations recorded by the source corpus.

We describe the DeepMind Kinetics human action video dataset. The dataset contains 400 human action classes, with at least 400 video clips for each action. Each clip lasts around 10s and is taken from a different YouTube video. The actions are human focussed and cover a broad range of classes including human-object interactions such as playing instruments, as well as human-human interactions such as shaking hands. We describe the statistics of the dataset, how it was collected, and give some baseline performance figures for neural network architectures trained and tested for human action classification on this dataset. We also carry out a preliminary analysis of whether imbalance in the dataset leads to bias in the classifiers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Scaling Autoregressive Video Models (2019, methodology): “Moving from a constrained to a real world setting, we next apply our models to the Kinetics dataset (Kay et al., 2017), a large scale action-recognition dataset consisting of YouTube videos.”
VideoBERT: A Joint Model for Video and Language Representation Learning (2019, methodology): “, [9, 19, 36, 7]) to train convolutional neural networks for video classification.”
Group Normalization (2018, methodology): “2014), and on 3D convolutional networks for Kinetics video classification (Kay et al. 2017).”
RareAct: A video dataset of unusual interactions (2020, background): “Many of the popular action recognition datasets such as Kinetics [6] or AVA [4] concentrate on frequently occurring actions, to obtain a large number of clips for each action class.”
Learning Video Representations from Textual Web Supervision (2020, methodology): “Our non-static dataset provides an additional advantage over large-scale labeled datasets (such as Kinetics [32]), in which videos can be deleted by their owners at any time.”
A Short Note on the Kinetics-700 Human Action Dataset (2019, methodology): “Full details can be found in the original publication [7].”
Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, methodology): “Carreira and Zisserman [10] proposed the Inception 3D (I3D) model by inflating all the 2D convolution filters and pooling kernels used by the Inception V1 architecture [29] into 3D convolutions and pre-training the model on the large-scale Kinetics human action video dataset [30].”
Multiview Transformers for Video Recognition (2022, methodology): “Figure 3 compares our proposed MTV to its single-view counterpart, ViViT Fac-torized Encoder (FE) [3] at every model scale on Kinetics 400.”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “The widely used Imagenet-1k [17] and Kinetics-400 [32] are used for evaluating the image and video classification tasks, respectively.”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “Among the literature, BEiT [70] and VIMPAC [71] are two relevant studies of masked visual modeling for image classification [42] and action recognition [37].”
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, methodology): “The clip-level motion information is captured by inflated 3D ResNeXt-101 [13, 50] pre-trained on Kinetics [21].”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, background): “A large number of architectural developments have been driven by action recognition on datasets such as Kinetics [26] where manual labelling has been relatively easier than obtaining textual descriptions for datasets.”
A Short Note on the Kinetics-700-2020 Human Action Dataset (2020, methodology): “Three editions have been released: Kinetics-400 [6], Kinetics-600 [1] and Kinetics-700 [2], with 400, 600 and 700 human action classes, respectively.”
ActBERT: Learning Global-Local Video-Text Representations (2020, methodology): “We initialized the weights of the ResNet-3D model from a pre-trained model on Kinetics [12].”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “For Video Embedder, we first use ResNet (He et al., 2016) pre-trained on ImageNet (Deng et al., 2009) and SlowFast (Feichtenhofer et al., 2019) pre-trained on Kinetics (Kay et al., 2017) to extract 2D and 3D visual features for each video frame.”
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, methodology): “[28] first built an action list by combining previous video datasets [24, 31, 53, 3, 62], and then searched the videos from YouTube for candidates, which eventually were filtered by Amazon Mechanical Turkers.”
Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity (2022, background): “UVO is a video instance segmentation dataset of YouTube videos (Kinetics400 [26]) with object masks exhaustively labeled.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “A typical raw video clip in our dataset lasts 8 minutes—significantly longer than the 10 second clips often studied in third-person video understanding [108].”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “Kinetics-700 [41,42] 2019 YouTube 1806 1 650k 10.”