Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification

By Zuxuan Wu, Xi Wang, Yu-Gang Jiang, Hao Ye, Xiangyang Xue. 2015

470 citations recorded by the source corpus.

Classifying videos according to content semantics is an important problem with a wide range of applications. In this paper, we propose a hybrid deep learning framework for video classification, which is able to model static spatial information, short-term motion, as well as long-term temporal clues in the videos. Specifically, the spatial and the short-term motion features are extracted separately by two Convolutional Neural Networks (CNN). These two types of CNN-based features are then combined in a regularized feature fusion network for classification, which is able to learn and utilize feature relationships for improved performance. In addition, Long Short Term Memory (LSTM) networks are applied on top of the two features to further model longer-term temporal clues. The main contribution of this work is the hybrid learning framework that can model several important aspects of the video data. We also show that (1) combining the spatial and the short-term motion features in the regularized fusion network is better than direct classification and fusion using the CNN with a softmax layer, and (2) the sequence-based LSTM is highly complementary to the traditional classification strategy without considering the temporal frame orders. Extensive experiments are conducted on two popular and challenging benchmarks, the UCF-101 Human Actions and the Columbia Consumer Videos (CCV). On both benchmarks, our framework achieves very competitive performance: 91.3% on the UCF-101 and 83.5% on the CCV.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding (2017, background): “In [52, 10], CNN layers are constructed to extract visual features while the followed recurrent layers are applied to handle temporal dynamics.”
Actions ~ Transformations (2015, methodology): “Motivated by the recent literatures [25, 48, 4] in using LSTM to model the temporal information of videos, we also conduct experiments using the LSTM baseline for comparison besides the Two Stream baseline.”
Jointly Modeling Embedding and Translation to Bridge Video and Language (2015, methodology): “These additions to the single memory cell enable LSTM to capture extremely complex and long-term temporal dynamics, which has also been applied to video classification [32].”
An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data (2016, background): “However, in comparisons to the holistic based approaches (Simonyan and Zisserman 2014; Wu et al. 2015; Zhu et al. 2016) which use all the frames, it loses some information.”
Online Human Action Detection using Joint Classification-Regression Recurrent Neural Networks (2016, methodology): “A hybrid deep learning framework is proposed for video classification [12], where LSTM networks are applied on top of the two types of CNN-based features related to the spatial and the short-term motion information.”
Temporal Segment Networks for Action Recognition in Videos (2017, methodology): “Several works have been trying to learn deep features and design effective ConvNet architectures for action recognition in videos [1], [4], [5], [15], [16], [23], [24], [52], [53], [54].”
Learning Long-Term Dependencies for Action Recognition with a Biologically-Inspired Deep Network (2016, background): “Yue-Hei et al. [44] and Donahue et al. [7] proposed their own recurrent networks respectively by connecting LSTMs to CNNs. Wu et al. [43] achieved the state-of-the-art performance by connecting CNNs and LSTMs under the hybrid deep learning framework.”
Fusing Multi-Stream Deep Networks for Video Classification (2015, methodology): “very recent studies attempted to use recurrent neural networks (RNN) to model long-term temporal information and achieved competitive performance [32, 46].”
Learning Activity Progression in LSTMs for Activity Detection and Early Detection (2016, background): “LSTM models are explored in [14, 26] for recognizing human actions.”
Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks (2015, background): “More recently, in addition to the CNN, researchers also adopted recurrent neural networks (RNN) to model the long-term temporal information in videos [16], [17], [18] and reported promising results.”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “Several other works [76], [93], [94] focused on the hybrid architectures combining two-stream 2D CNN and RNN models for HAR.”
A novel attention-based hybrid CNN-RNN architecture for sEMG-based gesture recognition (2018, background): “[31] designed a hybrid deep learning framework to extract spatial, short-term and long-term features which consist of two hybrid CNN-RNN architecture and a regularized fusion layer.”
Global Context-Aware Attention LSTM Networks for 3D Action Recognition (2017, background): “Recently, Long Short-Term Memory (LSTM) network [15] has been successfully applied to language modeling [46], RGB-based activity analysis [17, 68, 69, 61, 10, 21, 43, 30], and also 3D action recognition [11, 73, 27] due to its strong power in modeling sequential data.”