Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition

By Kensho Hara, Hirokatsu Kataoka, Yutaka Satoh. 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), 2017

710 citations recorded by the source corpus.

Convolutional neural networks with spatio-temporal 3D kernels (3D CNNs) have an ability to directly extract spatiotemporal features from videos for action recognition. Although the 3D kernels tend to overfit because of a large number of their parameters, the 3D CNNs are greatly improved by using recent huge video databases. However, the architecture of3D CNNs is relatively shallow against to the success of very deep neural networks in 2D-based CNNs, such as residual networks (ResNets). In this paper, we propose a 3D CNNs based on ResNets toward a better action representation. We describe the training procedure of our 3D ResNets in details. We experimentally evaluate the 3D ResNets on the ActivityNet and Kinetics datasets. The 3D ResNets trained on the Kinetics did not suffer from overfitting despite the large number of parameters of the model, and achieved better performance than relatively shallow networks, such as C3D. Our code and pretrained models (e.g. Kinetics and ActivityNet) are publicly available at https://github.com/kenshohara/3D-ResNets.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

View Paper: https://arxiv.org/abs/1708.07632

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet? (2017, methodology): “Unlike previous studies that examined only limited 3D ResNet architectures [9, 24], we examine not only deeper architectures but also some extended versions of ResNet.”
Unsupervised Learning From Video With Deep Neural Embeddings (2019, background): “ecture ˚and sampling strategy ˆ. Recent exploration in supervised action recognition has provided a variety of sensible choices for the form of ˚ . Although very complex network options are possible [4, 9], since this work is an initial exploration of the interplay between video processing architect”
Frequency Domain Compact 3D Convolutional Neural Networks (2020, methodology): “[10] pushed the classical 2D-ResNet [13] to three di-mension and achieved better performance than the relatively shallow C3D networks.”
Evolving Space-Time Neural Architectures for Videos (2018, neutral): “Approaches considering a video as a space-time volume have been particularly successful [1, 4, 30, 31], with a direct application of 3D CNNs to videos.”
Exploring Temporal Coherence for More General Video Face Forgery Detection (2021, background): “A 3D R50 [23] network structure, which employs spatio-temporal convolutions.”
DynamoNet: Dynamic Action and Motion Network (2019, neutral): “58.9 - C3D [17] 55.6 - 3D ResNet101 [17] 62.8 83.9 3D ResNext101 [17] 65.1 85.7 RGB-I3D [3] 68.4 88 STC-ResNet101 (16 frames) [6] 64.1 85.2 STC-ResNext101 (16 frames) [6] 66.2 86.5 STC-ResNext101 (32 frames) [6] 68 compared with the other 3D ConvNets who have provided results on the dataset.”
Clinically Applicable AI System for Accurate Diagnosis, Quantitative Measurements, and Prognosis of COVID-19 Pneumonia Using Computed Tomography (2020, methodology): “The detailed structure of the 3D classification network was shown in Figure S2B, adapted from 3D ResNet-18 (Hara et al., 2017).”
Low-Memory Neural Network Training: A Technical Report (2019, background): “For certain datasets, such as video [Hara et al., 2017] and high-resolution medical images [Qiu et al.”
FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling (2022, methodology): “1(a), existing methods usually regress quality scores with fixed features extracted from pre-trained networks for classification tasks [11,33,10] to alleviate memory shortage problem on GPUs instead of end-to-end training, preventing them from learning video-quality-related representations that…”
FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, background): “The astounding performance of deep neural networks for action recognition [5, 43, 58, 12, 42, 56] are obtained by densely applying 2D [56, 31, 58, 13] or 3D [40, 5, 18, 12] models over video frames.”
Progressive Teacher-Student Learning for Early Action Prediction (2019, background): “Recently, with the rise of deep learning, many deep learning based methods, including 3D CNN [39] [5] [12] [40] [13] and two stream CNN [36] [8] [46] etc.”
Holistic Large Scale Video Understanding (2019, neutral): “8 3D ResNext 101 (16 frames) [19] Kinetics ResNext101 90.”