Towards Good Practices for Very Deep Two-Stream ConvNets

By Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao. 2015

463 citations recorded by the source corpus.

Deep convolutional networks have achieved great success for object recognition in still images. However, for action recognition in videos, the improvement of deep convolutional networks is not so evident. We argue that there are two reasons that could probably explain this result. First the current network architectures (e.g. Two-stream ConvNets) are relatively shallow compared with those very deep models in image domain (e.g. VGGNet, GoogLeNet), and therefore their modeling capacity is constrained by their depth. Second, probably more importantly, the training dataset of action recognition is extremely small compared with the ImageNet dataset, and thus it will be easy to over-fit on the training dataset. To address these issues, this report presents very deep two-stream ConvNets for action recognition, by adapting recent very deep architectures into video domain. However, this extension is not easy as the size of action recognition is quite small. We design several good practices for the training of very deep two-stream ConvNets, namely (i) pre-training for both spatial and temporal nets, (ii) smaller learning rates, (iii) more data augmentation techniques, (iv) high drop out ratio. Meanwhile, we extend the Caffe toolbox into Multi-GPU implementation with high computational efficiency and low memory consumption. We verify the performance of very deep two-stream ConvNets on the dataset of UCF101 and it achieves the recognition accuracy of $91.4\%$.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, methodology): “A multi-scale cropping scheme is applied for fine-tuning and testing of all architectures [32].”
Generating Videos with Scene Dynamics (2016, neutral): “1% ImageNet Supervision [45]91.”
Pyramid Scene Parsing Network (2016, methodology): “To achieve this, we modify Caffe from [37] together with Method Mean IoU(%) Pixel Acc.”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, background): “Since then, many video classiﬁcation methods follow the same multi-stream 2D CNN design, and have made improvements in terms of new representations [25,26], different backbone architecture [27,28,29,17], fusion of the streams [30,31,32,33] and exploiting richer temporal structures [34,35,36].”
Delving Deeper into Convolutional Networks for Learning Video Representations (2015, methodology): “Our model is on part with Wang et al. (2015b) that obtain state-of-art results using both RGB and ﬂow streams which obtains 90 .”
Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition (2017, methodology): “In addition to the positions, we also select the spatial scales of each sample to perform multi-scale cropping [21].”
Convolutional Two-Stream Network Fusion for Video Action Recognition (2016, methodology): “3.5) that samples from the image corners and its centre [35].”
Actions ~ Transformations (2015, methodology): “By replacing the base network with VGG-16 architecture, [53] obtains 91 .”
Long-Term Temporal Convolutions for Action Recognition (2016, positive): “Consistent with previous results obtained with handcrafted representations, motion-based CNNs typically outperform CNN representations learned for RGB inputs [6], [23].”
Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet? (2017, methodology): “The procedure used is the same as [28].”
Temporal Segment Networks for Action Recognition in Videos (2017, neutral): “VGG-M [69] (from [1]) 73:0% 83:7% 86:9% GoogLeNet [9] (from [70]) 75:3% 85:8% 89:3% VGGNet-16 [8] (from [70]) 78:4% 87:0% 91:4% ResNet-152 [31] (from [71]) 83:4% 87:2% 91:8% BN-Inception [62] 85:0% 88:3% 92:4%”
Unsupervised Learning using Sequential Verification for Action Recognition (2016, methodology): “We use the improved data augmentation scheme (different aspect-ratio, fixed crops) from [72] for all our methods and baselines.”
MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, background): “In contrast to traditional multi-clip evaluation approaches [54, 67] which also reduce memory, a stream buffer carries over temporal dependencies between consecutive non-overlapping subclips by caching feature maps at subclip boundaries.”
ActionVLAD: Learning Spatio-Temporal Aggregation for Action Classification (2017, result): “Similar to the two-stream approaches [57, 58] that typically use multiple crops and ﬂips of the video frames at test time to average predictions over, we can also perform Ac-tionVLAD pooling over multiple crops of the input frames.”
RPAN: An End-to-End Recurrent Pose-Attention Network for Action Recognition in Videos (2017, methodology): “Since we follow [46, 47] to process two streams separately, we henceforth describe the”
Factors in Finetuning Deep Model for Object Detection with Long-Tail Distribution (2016, background): “Deep learning is found to be effective in many vision tasks [38, 4, 40, 39, 21, 24, 23, 49, 19, 34, 33, 7, 48, 31].”
Learning Multimodal Attention LSTM Networks for Video Captioning (2017, background): “[26] extract deep feature and conducted trajectory constrained pooling to aggregate convolutional feature as video representations.”
Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition (2017, background): “been proposed by considering video action recognition as a classification task [5, 43, 23, 50, 11, 10, 9, 41, 42, 33, 29].”
BSN: Boundary Sensitive Network for Temporal Action Proposal Generation (2018, background): “In recent years, convolutional networks are widely adopted in many works [10,33,35,41] and have achieved great performance.”