Compressed Video Action Recognition

By Chao-Yuan Wu, Manzil Zaheer, Hexiang Hu, Raghavan Manmatha, Alex Smola, Philipp Krahenbuhl. 2017

371 citations recorded by the source corpus.

Training robust deep video representations has proven to be much more challenging than learning deep image representations. This is in part due to the enormous size of raw video streams and the high temporal redundancy; the true and interesting signal is often drowned in too much irrelevant data. Motivated by that the superfluous information can be reduced by up to two orders of magnitude by video compression (using H.264, HEVC, etc.), we propose to train a deep network directly on the compressed video. This representation has a higher information density, and we found the training to be easier. In addition, the signals in a compressed video provide free, albeit noisy, motion information. We propose novel techniques to use them effectively. Our approach is about 4.6 times faster than Res3D and 2.7 times faster than ResNet-152. On the task of action recognition, our approach outperforms all the other methods on the UCF-101, HMDB-51, and Charades dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Endocrinology, Diabetes and Metabolism

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Endocrinology, Diabetes and Metabolism

Citations in the corpus

AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, neutral): “Method modality mAP 2-Strm. [20] (from [18]) RGB+Flow 18.6 Asyn-TF [18] RGB+Flow 22.4 CoViAR [28] Compressed 21.9 MultiScale TRN [33] RGB 25.2 I3D [3] RGB 32.9 I3D [3] (from [25]) RGB 35.5 I3D-NL [25] RGB 37.5 STRG [26] RGB 39.7 LFB [27] RGB 42.5 SlowFast [6] RGB+RGB 45.”
Long-Term Feature Banks for Detailed Video Understanding (2018, neutral): “4 CoViAR [54] R50 Compressed 21.”
Cross-Modal and Hierarchical Modeling of Video and Text (2018, background): “Deep learning has brought significant improvement to video understanding [30, 33, 6, 38, 44, 41] on large-scale action recognition datasets [9, 31, 14] in the past decade.”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, methodology): “Due to the high computational demands of optical ﬂow extraction, several attempts in compressed video recognition [8, 48, 63, 71] have begun to utilize more efﬁcient motion vectors as an alternative to represent motions and have shown promising performance.”
Variable-rate discrete representation learning (2021, background): “Some models operate directly on compressed representations to achieve a similar effect (Nash et al., 2021), though usually some topological structure is preserved (Gueguen et al., 2018; Wu et al., 2018a).”
Multi-teacher Knowledge Distillation for Compressed Video Action Recognition on Deep Neural Networks (2019, methodology): “For video-level tasks, CoViAR [10] uses a sparse sampling strategy [3] on an input video, where the samples distribute uniformly along the temporal dimension, aggregating information from the samples during training.”
Video Compression through Image Interpolation (2018, background): “Video compression even helps better understand and parse videos using deep neural networks [32].”
Multi-Fiber Networks for Video Recognition (2018, background): “In a very interesting recent approach, CoViAR [18] further reduces computations to 4.2 GFLOPs per frame in average, by directly using the motion information from compressed frames and sharing motion features across frames.”
RubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition (2020, neutral): “Additionally, our work remains complementary to literature on neural network compression [30], so while our proposed method saves substantially on model size and computation cost, it is possible that application of such techniques can lead to further gains.”
LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition (2019, neutral): “While there is plethora of work focusing on designing robust models for video classification, limited efforts have been made on efficient video recognition [42, 36, 4, 41, 38, 5, 19, 43].”
AdaFrame: Adaptive Frame Selection for Fast Video Recognition (2018, methodology): “Most existing work focuses on learning robust video representations to boost accuracy [24, 29, 19, 28], while limited effort has been devoted to improving efficiency [31, 38].”
DVC: An End-To-End Deep Video Compression Framework (2018, methodology): “Meanwhile, the techniques in video compression are also helpful for action recognition [41] and model compression [16].”
Hidden Two-Stream Convolutional Networks for Action Recognition (2017, background): “sub-optimal. To solve the above problems, researchers have proposed various methods other than optical ow to capture motion information in videos. For example, new representations like motion vectors [27,33] and RGB image dierence [25] or architectures like recurrent neural networks (RNN) [16] and 3”
X3D: Expanding Architectures for Efficient Video Recognition (2020, neutral): “[2,5,9, 11, 13, 18, 28, 35, 38, 42, 44, 55, 56, 63, 65, 66, 72, 76, 84–86].”
SCSampler: Sampling Salient Clips From Video for Efficient Action Recognition (2019, methodology): “We do not present results for the large ResNet-152 model that was used in [52], since it adds a cost of 3 GFLOPS per clip which far exceeds the computational budget of our application.”
SlowFast Networks for Video Recognition (2018, methodology): “model pretrain mAP GFLOPs×views CoViAR, R-50 [55] ImageNet 21.”
Listen to Look: Action Recognition by Previewing Audio (2019, background): “Some work balances the accuracy-efficiency trade-off by using compressed video representations [68, 49] or designing efficient network architectures [72, 81, 10, 57].”