Real-Time Action Recognition with Enhanced Motion Vector CNNs

By Bowen Zhang, Limin Wang, Zhe Wang, Yu Qiao, Hanli Wang. 2016

439 citations recorded by the source corpus.

The deep two-stream architecture [23] exhibited excellent performance on video based action recognition. The most computationally expensive step in this approach comes from the calculation of optical flow which prevents it to be real-time. This paper accelerates this architecture by replacing optical flow with motion vector which can be obtained directly from compressed videos without extra calculation. However, motion vector lacks fine structures, and contains noisy and inaccurate motion patterns, leading to the evident degradation of recognition performance. Our key insight for relieving this problem is that optical flow and motion vector are inherent correlated. Transferring the knowledge learned with optical flow CNN to motion vector CNN can significantly boost the performance of the latter. Specifically, we introduce three strategies for this, initialization transfer, supervision transfer and their combination. Experimental results show that our method achieves comparable recognition performance to the state-of-the-art, while our method can process 390.7 frames per second, which is 27 times faster than the original two-stream method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, positive): “[170] accelerated this two stream structure by replacing optical flow with motion vector which can be obtained directly from compressed videos without extra calculation.”
Long-Term Temporal Convolutions for Action Recognition (2016, positive): “Similar findings have been recently confirmed by [26], where the authors transfer knowledge from high quality optical flow algorithms to motion vector encoding representation.”
Deep Mutual Learning (2017, methodology): “[29] for action recognision: the more expensive optical flow field is treated as privileged information and an optical flow CNN is used to teach a motion vector CNN.”
Temporal Segment Networks for Action Recognition in Videos (2017, methodology): “Two-Stream 3D ConvNet [68] - 90:2% 246 RGB+EMV [17] - 86.”
Cross-Modal and Hierarchical Modeling of Video and Text (2018, background): “Deep learning has brought significant improvement to video understanding [30, 33, 6, 38, 44, 41] on large-scale action recognition datasets [9, 31, 14] in the past decade.”
Efficient Video Classification Using Fewer Frames (2019, background): “Some work on video-based action recognition [41] tries to accelerate processing in a two-stream CNN architecture by transferring knowledge from motion modality to optical modality.”
Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition (2021, methodology): “Compared with RGB video based methods which require high-cost computing resources to process pixel-level information in RGB image or temporal optical flow (Tran et al. 2015; Simonyan and Zisserman 2014; Zhang et al. 2016; Li et al. 2020), skeleton”
Actionness Estimation Using Hybrid Fully Convolutional Networks (2016, background): “Recently, several attempts have been made in applying CNN for action recognition from videos [30, 17, 40, 50].”
StNet: Local and Global Spatial-Temporal Modeling for Action Recognition (2018, background): “Action recognition in videos has received significant research attention in the computer vision and machine learning community (Karpathy et al. 2014; Wang and Schmid 2013; Wang, Qiao, and Tang 2016; Simonyan and Zisserman 2014; Fernando et al. 2015; Wang et al. 2016; Qiu, Yao, and Mei 2017; Carreira”
SCSampler: Sampling Salient Clips From Video for Efficient Action Recognition (2019, background): “For example, video summarization [18, 19, 29, 37, 56, 57, 58] and the automatic production of sport highlights [30, 31] entail creating a much shorter version of the original video by concatenating a small set of snippets corresponding to the most informative or exciting moments.”
A Survey on Deep Learning Based Approaches for Action and Gesture Recognition in Image Sequences (2017, background): “[134] 2016 CNN 2 CNNs (spatial + temporal) 86.”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “To tackle the high computational cost of computing accurate optical flow, some works [60] aimed to mimic the knowledge of the flow stream during training, in order to avoid the usage of optical flow during testing.”
A Unified Framework for Multi-Modal Isolated Gesture Recognition (2018, background): “Unlike previous works (Ng et al. 2015; Zhang et al. 2016; Wang and Schmid 2013) that focus purely on RGB modality, we try to build a unified framework that exploits the inherent advantages of multi-modality modelling, namely RGB stream, depth stream, and saliency stream.”
Temporal Action Detection with Structured Segment Networks (2017, background): “Action recognition has been extensively studied in the past few years [21, 46, 39, 44, 49, 51, 57].”
Multi-teacher Knowledge Distillation for Compressed Video Action Recognition on Deep Neural Networks (2019, background): “Some approaches explored other robust deep video representations [9, 10], such as CoViAR [10], to train a deep network directly on the compressed video.”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, background): “Due to the high computational demands of optical ﬂow extraction, several attempts in compressed video recognition [8, 48, 63, 71] have begun to utilize more efﬁcient motion vectors as an alternative to represent motions and have shown promising performance.”
Self-Referenced Deep Learning (2018, background): “elate distillation to information learning theory for which a teacher provides privileged information (e.g. sample explanation) to a student in order to facilitate fast learning [33,43]. Zhang et al. [51] exploited this idea for video based action recognition by considering the computationally expen”
Dynamic Sampling Networks for Efficient Action Recognition in Videos (2020, neutral): “Method Input Clips mAP RGB+EMV [57] RGB+FLOW - 61.”