Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?

By Kensho Hara, Hirokatsu Kataoka, Yutaka Satoh. 2017

2,232 citations recorded by the source corpus.

The purpose of this study is to determine whether current video datasets have sufficient data for training very deep convolutional neural networks (CNNs) with spatio-temporal three-dimensional (3D) kernels. Recently, the performance levels of 3D CNNs in the field of action recognition have improved significantly. However, to date, conventional research has only explored relatively shallow 3D architectures. We examine the architectures of various 3D CNNs from relatively shallow to very deep ones on current video datasets. Based on the results of those experiments, the following conclusions could be obtained: (i) ResNet-18 training resulted in significant overfitting for UCF-101, HMDB-51, and ActivityNet but not for Kinetics. (ii) The Kinetics dataset has sufficient data for training of deep 3D CNNs, and enables training of up to 152 ResNets layers, interestingly similar to 2D ResNets on ImageNet. ResNeXt-101 achieved 78.4% average accuracy on the Kinetics test set. (iii) Kinetics pretrained simple 3D architectures outperforms complex 2D architectures, and the pretrained ResNeXt-101 achieved 94.5% and 70.2% on UCF-101 and HMDB-51, respectively. The use of 2D CNNs trained on ImageNet has produced significant progress in various tasks in image. We believe that using deep 3D CNNs together with Kinetics will retrace the successful history of 2D CNNs and ImageNet, and stimulate advances in computer vision for videos. The codes and pretrained models used in this study are publicly available1.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, methodology): “However, SGD is preferred for 3D CNN architectures [11,1,7,27,3].”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “3D features are extracted with the Kinetics [4] pre-trained ResNeXt-101 16-frames model [12] to obtain 1.”
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, methodology): “From the results, we can conclude that, for all types of questions, the best performance is from using ResNet as an appearance feature along with I3D ResNeXt as a motion feature (Res+I3D).”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, methodology): “For a long time this space was dominated by spatio-temporal CNNs such as I3D [8], 3D ResNets [18], S3D [64] or ‘R(2+1)D’ CNNs [55].”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, methodology): “For 3D feature extraction, we employ ResNeXt-101 (Hara et al., 2018) that pre-trained on Kinetics to extract 3D features.”
Benchmarking Deep Learning Models and Automated Model Design for COVID-19 Detection with Chest CT Scans (2020, methodology): “the 3D models, we use DenseNet3D121 [17], R2Plus1D [18], MC3 18 [18], ResNeXt3D101 [17], PreAct ResNet3D [17], and ResNet3D series [17] (ResNet3D10, ResNet3D18, ResNet3D34, ResNet3D50, ResNet3D101, ResNet3D152, and ResNet3D200).”
Video Action Recognition Via Neural Architecture Searching (2019, neutral): “, 3D-ResNet [19], C3D network [20], and STC-ResNet [21].”
Video Representation Learning by Dense Predictive Coding (2019, methodology): “We choose to use a 3D-ResNet similar to [9] as the encoder f(.”
Hierarchical Conditional Relation Networks for Video Question Answering (2020, methodology): “In our experiments, v i,j are the pool5 output of ResNet [9] features and f i are derived by ResNeXt-101 [38, 8].”
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, methodology): “The gap between the fusion result and the supervised result [14]”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, methodology): “5 features per second using a ResNeXt-101 model [34] pretrained on Kinetics [35].”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, methodology): “We extract 3D features using Kinetics (Carreira and Zisserman 2017) pre-trained ResNeXt-101 16- frames (Hara, Kataoka, and Satoh 2018) at a rate of 24 frames per second.”
When Video Classification Meets Incremental Classes (2021, methodology): “After that, a large number of improvements have been made based on this work, eg, using deeper networks [16], preventing networks from overfitting [50], fusing two-stream features better [12–14, 52], and multi-stream network [57].”
Unsupervised Learning From Video With Deep Neural Embeddings (2019, background): “R3D networks of [29] and the 3DResNets of [10].”
Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video (2020, background): “Hara et al. [32] studied whether 3D CNNs based on standard 2D ResNet [33] architectures could be exploited for action recognition from video.”
MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, methodology): “We perform the search on different FLOPs targets to produce a family of models ranging from MobileNetV3-like sizes up to the sizes of ResNet3D-152 [24, 23].”
Spatiotemporal Contrastive Video Representation Learning (2020, methodology): “Similarly, state-of-the-art neural networks for video understanding [63, 9, 31, 71, 17, 16] often extend 2D convolutional neural networks [33, 36] for images along the temporal dimension.”
In the Eye of the Beholder: Gaze and Actions in First Person Video (2020, background): “explored, including mixed 2D and 3D networks [45], [46], grouped or depthwise 3D convolutions [24], [47], [48], [49], and the use of network architecture search [50].”
VA-RED2: Video Adaptive Redundancy Reduction (2021, neutral): “…(Karpathy et al., 2014; Simonyan & Zisserman, 2014; Chéron et al., 2015; Feichtenhofer et al., 2017; Gkioxari & Malik, 2015; Wang et al., 2016; Zhou et al., 2018a; Lin et al., 2019; Fan et al., 2019) or 3D CNNs (Tran et al., 2015; Carreira & Zisserman, 2017; Hara et al., 2018; Tran et al., 2018).”
Discriminability Distillation in Group Representation Learning (2020, methodology): “We select three clip-based action recognition baseline method, the 3D-ResNet-50 [22], SlowFast-50 [13] and R(2+1)D-50 [51].”