Beyond Gaussian Pyramid: Multi-skip Feature Stacking for action recognition

By Zhenzhong Lan, Ming Lin, Xuanchong Li, Alexander G. Hauptmann, Bhiksha Raj. 2014

301 citations recorded by the source corpus.

Most state-of-the-art action feature extractors involve differential operators, which act as highpass filters and tend to attenuate low frequency action information. This attenuation introduces bias to the resulting features and generates ill-conditioned feature matrices. The Gaussian Pyramid has been used as a feature enhancing technique that encodes scale-invariant characteristics into the feature space in an attempt to deal with this attenuation. However, at the core of the Gaussian Pyramid is a convolutional smoothing operation, which makes it incapable of generating new features at coarse scales. In order to address this problem, we propose a novel feature enhancing technique called Multi-skIp Feature Stacking (MIFS), which stacks features extracted using a family of differential filters parameterized with multiple time skips and encodes shift-invariance into the frequency space. MIFS compensates for information lost from using differential operators by recapturing information at coarse scales. This recaptured information allows us to match actions at different speeds and ranges of motion. We prove that MIFS enhances the learnability of differential-based features exponentially. The resulting feature matrices from MIFS have much smaller conditional numbers and variances than those from conventional methods. Experimental results show significantly improved performance on challenging action recognition and event detection tasks. Specifically, our method exceeds the state-of-the-arts on Hollywood2, UCF101 and UCF50 datasets and is comparable to state-of-the-arts on HMDB51 and Olympics Sports datasets. MIFS can also be used as a speedup strategy for feature extraction with minimal or no accuracy cost.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Delving Deeper into Convolutional Networks for Learning Video Representations (2015, background): “classification (Simonyan & Zisserman, 2014b), CNNs did not yield large improvement over these traditional methods (Lan et al., 2014) highlighting the difficulty of learning video representations even with large training dataset.”
Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection (2017, background): “Many traditional works in the field of action recognition focused on designing features to discriminate action classes [19, 44, 5, 18, 17].”
Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks (2017, background): “Two kinds of motion representations can be identified: Lagrangian motion [48, 18, 31, 50, 30, 56] and Eulerian motion [2, 25, 60, 51, 52, 1].”
Unsupervised Learning of Long-Term Motion Dynamics for Videos (2017, background): “The past few years have seen great progress on activity recognition on short clips [13, 51, 28, 38, 40].”
Towards Good Practices for Very Deep Two-Stream ConvNets (2015, background): “…without using these [20], without using our proposed good practices) Method Year Accuracy iDT+FV [16] 2013 85.9% iDT+HSV [10] 2014 87.9% MIFS+FV [8] 2015 89.1% TDD+FV [19] 2015 90.3% DeepNet [6] 2014 63.3% Two-stream [12] 2014 88.0% Two-stream+LSTM [9] 2015 88.6% Very deep two-stream 2015 91.4%…”
Actions ~ Transformations (2015, background): “More recently, trajectory based approaches [51, 31, 29, 50, 18, 32, 25] have shown signiﬁcant improvement in action recognition.”
Action Recognition using Visual Attention (2015, neutral): “(Srivastava et al., 2015) 44.0 - DL-SFA (Sun et al., 2014) - 48.1 Two-stream ConvNet (Simonyan & Zisserman, 2014) 59.4 - VideoDarwin (Fernando et al., 2015) 63.7 73.7 Multi-skIp Feature Stacking (Lan et al., 2014) 65.1 68.0 Traditional+Stacked Fisher Vectors (Peng et al., 2014) 66.8 - Objects+Tr”
Long-Term Temporal Convolutions for Action Recognition (2016, neutral): “2 [32] IDT+MIFS 89.”
Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification (2015, background): “Srivastava et al. [36] 84.3% Ye et al. [50] 64.0% Wang et al. [44] 85.9% Jhuo et al. [12] 64.0% Tran et al. [40] 86.7% Ma et al. [27] 63.4% Simonyan et al. [33] 88.0% Liu et al. [25] 68.2% Lan et al. [22] 89.1% Wu et al. [47] 70.6% Zha et al. [52] 89.6% / Ours 91.3% Ours 83.5% Table 3: Comparison wi”
VideoLSTM convolves, attends and flows for action recognition (2016, background): “ed work. 2 Related work The literature on action recognition in video is vast and too broad for us to cover here completely. We reckon and value the impact of traditional video representations, e.g., [3,4,5,19,20,21,22], and recent mixtures of shallow and deep encodings e.g., [23,24]. Here we focus ”
When Video Classification Meets Incremental Classes (2021, background): “With the rise of deep learning, CNNs dominated the video understanding literature recently [3, 21, 27, 30, 36, 44, 54, 56], instead of hand-crafted features [1, 26, 35, 46, 47].”
Asynchronous Temporal Fields for Action Recognition (2016, background): “Understanding activities and actions has an extensive history [32, 59, 22, 17, 23, 2, 26, 56, 29, 21].”
Action Recognition by Learning Deep Multi-Granular Spatio-Temporal Video Representation (2016, methodology): “IDT: improved dense trajectory [29]; MIFS: Multi-skip Feature Stacking [13]; LRCN: Long-term Recurrent Convolutional Networks [3]; C3D: Convolutional 3D [27]; TDD: Trajectory-pooled Deep-convolutional Descriptor [30].”
Learning Long-Term Dependencies for Action Recognition with a Biologically-Inspired Deep Network (2016, methodology): “These results are much better than traditional iDT [41] and MIFS [23].”
Large-scale Isolated Gesture Recognition using pyramidal 3D convolutional networks (2016, methodology): “[28] proposed a multi-skip feature stacking technique which stacks features extracted using a family of differential filters parameterized with multiple time skips.”
Expert knowledge-infused deep learning for automatic lung nodule detection (2018, unknown): “…of computer vision, to overcome the challenges of the limited existing datasets and the difficulty for network to learn various types of robust features simultaneously, some researchers have proposed methods to incorporate the hand-crafted features in combination with CNN based models ([22–27]).”
Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs (2016, background): “Impressive progress has been reported in recent literature for action recognition [42, 28, 2, 3, 39, 40, 24, 18, 31, 44, 13, 37].”