ConvNet Architecture Search for Spatiotemporal Feature Learning

By Du Tran, Jamie Ray, Zheng Shou, Shih-Fu Chang, Manohar Paluri. 2017

417 citations recorded by the source corpus.

Learning image representations with ConvNets by pre-training on ImageNet has proven useful across many visual understanding tasks including object detection, semantic segmentation, and image captioning. Although any image representation can be applied to video frames, a dedicated spatiotemporal representation is still vital in order to incorporate motion patterns that cannot be captured by appearance based models alone. This paper presents an empirical ConvNet architecture search for spatiotemporal feature learning, culminating in a deep 3-dimensional (3D) Residual ConvNet. Our proposed architecture outperforms C3D by a good margin on Sports-1M, UCF101, HMDB51, THUMOS14, and ASLAN while being 2 times faster at inference time, 2 times smaller in model size, and having a more compact representation.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet? (2017, background): “Conventional 3D CNN architectures trained on Kinetics are still relatively shallow (10 [16], 22 [2], and 34 [9, 24] layers).”
Spatio-Temporal Channel Correlation Networks for Action Classification (2018, methodology): “Our STC-ResNext101 models outperform the 3D-ResNet [5], Inception3D and C3D [2] on both UCF101 and HMDB51 by 95.8% and 72.6% respectively.”
Action Search: Spotting Actions in Videos and Its Application to Temporal Action Localization (2017, methodology): “More recently, end-to-end methods have proven to boost the performance of two-stage approaches, demonstrating the importance of jointly optimizing the feature extraction and detection process [3,41,43,46].”
FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, methodology): “Interestingly, FrameExit outperforms 3D models [41, 7] in this task.”
Differentiable Architecture Search with Ensemble Gumbel-Softmax (2019, background): “Recently, discovering neural architecture automatically has raised great interest in both academia and industry (Bello et al., 2016; Baker et al., 2017; Brock et al., 2017; Tran et al., 2017; Suganuma et al., 2018; Veit & Belongie, 2018).”
A Scalable Hierarchical Gaussian Process Classifier (2019, methodology): “Res3D features are extracted from res5b layer of Res3D net which is trained on Sports-1M dataset and fine-tuned on ucf101 [51].”
DATA: Differentiable ArchiTecture Approximation (2019, neutral): “, evolution-based NAS [13, 18, 26, 41, 43, 44, 45, 47], reinforcement learning-based NAS [2, 3, 21, 42, 56, 59, 60], and gradient-based NAS [11, 34, 35, 46, 53], which has achieved significant successes in a multitude of fields, including image classification [4, 12, 21, 30, 31, 34, 44, 53, 60], sem”
Co-Training of Audio and Video Representations from Self-Co-Training of Audio and Video Representations from Self-Supervised Temporal Synchronization Supervised Temporal Synchronization (2018, methodology): “Our video subnetwork is based on a mixed convolution (MCx) family of architectures [16, 17].”
PoTion: Pose MoTion Representation for Action Recognition (2018, background): “CNN architectures are either based on spatio-temporal convolutions [39, 40], recurrent neural networks [8] or two-stream architectures [32, 43].”
Multimodal Explanations by Predicting Counterfactuality in Videos (2018, methodology): “Specifically, we dealt with C3D-resnet [29, 12] as the target classifier in the experiments, which is based on the spatio-temporal convolution [28] with residual architecture [13].”
Rethinking Spatiotemporal Feature Learning For Video Understanding (2017, methodology): “This is equivalent to what [47] call “MC” networks, which stands for “mixed 3D-2D convolution”.”
BSN: Boundary Sensitive Network for Temporal Action Proposal Generation (2018, methodology): “To guarantee the experiment effectiveness, instead of two-stream network, here we adopt C3D network [36] trained on Sports-1M dataset [24] for video features encoding.”
Compressed Video Action Recognition (2017, methodology): “Nonetheless, it results in an explosion of parameters and computation time and only marginally improves the performance [38].”
ECO: Efficient Convolutional Network for Online Video Understanding (2018, methodology): “3D-Net: For the 3D network H3D we adopt several layers of 3D-Resnet18 [32], which is an efficient architecture used in many video classification works [32,15].”
Appearance-and-Relation Networks for Video Classification (2017, methodology): “Second , comparing two-stream C3D-ResNet18 with ARTNet-ResNet18, we notice that our proposed ARTNet is still able to yield a slightly better performance ( 21 .”