Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
By João Carreira, Andrew Zisserman. 2017
9,734 citations recorded by the source corpus.
The paucity of videos in current action classification datasets (UCF-101 and HMDB-51) has made it difficult to identify good video architectures, as most methods obtain similar performance on existing small-scale benchmarks. This paper re-evaluates state-of-the-art architectures in light of the new Kinetics Human Action Video dataset. Kinetics has two orders of magnitude more data, with 400 human action classes and over 400 clips per class, and is collected from realistic, challenging YouTube videos. We provide an analysis on how current architectures fare on the task of action classification on this dataset and how much performance improves on the smaller benchmark datasets after pre-training on Kinetics. We also introduce a new Two-Stream Inflated 3D ConvNet (I3D) that is based on 2D ConvNet inflation: filters and pooling kernels of very deep image classification ConvNets are expanded into 3D, making it possible to learn seamless spatio-temporal feature extractors from video while leveraging successful ImageNet architecture designs and even their parameters. We show that, after pre-training on Kinetics, I3D models considerably improve upon the state-of-the-art in action classification, reaching 80.2% on HMDB-51 and 97.9% on UCF-101.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Endocrinology, Diabetes and Metabolism
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Trevor Darrell
- Quoc V. Le
- Yoshua Bengio
- Dacheng Tao
- Sergey Levine
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Endocrinology, Diabetes and Metabolism
- Christoph Feichtenhofer
- Limin Wang
- Dima Damen
- Luc Van Gool
- Lorenzo Torresani
- Cordelia Schmid
- Rohit Girdhar
- Yu Qiao
Citations in the corpus
- Rethinking ImageNet Pre-Training (2018, background): “Pre-training has enabled state-of-the-art results on many tasks, including object detection [9, 8, 36], image segmentation [29, 13], and action recognition [42, 4].”
- Exploring the Limits of Weakly Supervised Pretraining (2018, background): “Examples include: object detection [1,4], semantic segmentation [5,6], human pose estimation [7,8], video recognition [9], monocular depth estimation [10], and so on.”
- Scaling Autoregressive Video Models (2019, methodology): “The conceptual simplicity of our model is in line with recent approaches to video classification that models videos by means of 3D convolutions [5, 43] or spatiotemporal self-attention [15].”
- Learning Representations by Maximizing Mutual Information Across Views (2019, methodology): “, [Devlin et al., 2019, Logeswaran and Lee, 2018, Sermanet et al., 2017, Dwibedi et al., 2018]. Some seminal work on self-supervised learning for computer vision involves predicting spatial structure or color information that has been procedurally removed from the data. E.g., Doersch et al. [2015] a”
- Group Normalization (2018, methodology): “We experiment with Inflated 3D (I3D) convolutional networks (Carreira and Zisserman 2017).”
- Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, methodology): “sults on dierent architectures, and comparison with state-of-the-art sections are presented, respectively. 4.1 Dataset Four datasets are relevant for our study: HMDB51 [16], UCF101 [25], Kinetics400 [1] and IG65M [9] datasets. HMDB51 consists of ˘7k clips with 51 classes whereas UCF101 includes ˘13k”
- Learning Video Representations from Textual Web Supervision (2020, methodology): “We choose these two architectures because R3D-50 provides high capacity and performance, while S3D-G offers lower computational cost while still outperforming well-known architectures such as I3D [9].”
- End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, positive): “We compare to a randomly initialized I3D and one inflated from an Inception network pretrained on ImageNet [15].”
- A Short Note on the Kinetics-700 Human Action Dataset (2019, methodology): “As a baseline model we used I3D [3], with standard RGB videos as input (no optical flow).”
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “3D features are extracted with the Kinetics [4] pre-trained ResNeXt-101 16-frames model [12] to obtain 1.5 features per second.”
- Objects as Points (2019, background): “Object detection powers many vision tasks like instance segmentation [7, 21, 32], pose estimation [3, 15, 39], tracking [24,27], and action recognition [5].”
- Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, methodology): “Carreira and Zisserman [10] proposed the Inception 3D (I3D) model by inflating all the 2D convolution filters and pooling kernels used by the Inception V1 architecture [29] into 3D convolutions and pre-training the model on the large-scale Kinetics human action video dataset [30].”
- Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, background): “Nevertheless, unsupervised networks are still significantly lower performing than their supervised counterparts, and are rarely used in real-world applications [7, 51, 8].”
- All in One: Exploring Unified Video-Language Pre-Training (2022, background): “Several classic ideas including sparse sampling [49], 3D-type operations [5, 36] are proposed for temporal modeling in both convolution and Transformer architectures.”
- Multiview Transformers for Video Recognition (2022, background): “We report accuracies when averaging predictions across multiple spatio-temporal crops, as standard practice [3, 6, 10, 23].”
- NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “From the results, we can conclude that, for all types of questions, the best performance is from using ResNet as an appearance feature along with I3D ResNeXt as a motion feature (Res+I3D).”
- Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, methodology): “For a long time this space was dominated by spatio-temporal CNNs such as I3D [8], 3D ResNets [18], S3D [64] or ‘R(2+1)D’ CNNs [55].”