Co-Occurrence Feature Learning for Skeleton Based Action Recognition Using Regularized Deep LSTM Networks

By Wentao Zhu, Cuiling Lan, Junliang Xing, Wenjun Zeng, Yanghao Li, Li Shen, Xiaohui S. Xie. 2016

926 citations recorded by the source corpus.

Skeleton based action recognition distinguishes human actions using the trajectories of skeleton joints, which provide a very good representation for describing actions. Considering that recurrent neural networks (RNNs) with Long Short-Term Memory (LSTM) can learn feature representations and model long-term temporal dependencies automatically, we propose an end-to-end fully connected deep LSTM network for skeleton based action recognition. Inspired by the observation that the co-occurrences of the joints intrinsically characterize human actions, we take the skeleton as the input at each time slot and introduce a novel regularization scheme to learn the co-occurrence features of skeleton joints. To train the deep LSTM network effectively, we propose a new dropout algorithm which simultaneously operates on the gates, cells, and output responses of the LSTM neurons. Experimental results on three human action recognition datasets consistently demonstrate the effectiveness of the proposed model.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “Recently, deep learning based-approaches have been proposed for 3D human activity analysis [9], [59], [60].”
Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos (2016, methodology): “The work of [89] introduced an internal dropout mechanism applied to LSTM gates for stronger regularization in the RNN-based 3D action learning network.”
Space-Time Representation of People Based on 3D Skeletal Data: A Review (2016, methodology): “[188] RNN with LSTM Conc Lowlv Deep X Chen et al.”
Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, methodology): “With the advent of cheap and easy-to-use depth sensors, such as Kinect [8], 3D skeleton-based human action recognition became a popular research domain [26, 27], and a series of hand-crafted features [28–37] and deep learning-based approaches [18, 19, 38–44] have been proposed.”
Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks (2017, methodology): “They have been successfully applied to language modeling [18], RGB based video analysis [19]–[27], and also skeleton based action recognition [12], [28], [29].”
Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates (2017, methodology): “An internal dropout [59] technique within the LSTM unit was also introduced in [58].”
Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks (2017, methodology): “[55] propose a fully connected deep LSTM network with regularization terms to learn co-occurrence features of joints.”
Interpretable 3D Human Action Analysis with Temporal Convolutional Networks (2017, background): “The work of [38] leverages on similar intuition that co-occurrence of joints is a strong discriminative feature for human action recognition.”
Learning and Refining of Privileged Information-Based RNNs for Action Recognition from Depth Sequences (2017, methodology): “RNN-based approaches become the dominant solution [61, 42, 9, 27] for action recognition from depth sequence recently.”
View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data (2017, methodology): “A general treatment employs a pre-processing step to transform the 3D joint coordinates from the camera coordinate system to a personcentric coordinate system by placing the body center at the origin, followed by rotating the skeleton such that the body plane is parallel to the (x, y)-plane, to make”
PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding (2017, methodology): “For the classifier, we choose three stacked bidirectional LSTM (BLSTM) network and SVM motivated by the effectiveness of LSTM models [57] and the agility of SVM classifier.”
A New Representation of Skeleton Sequences for 3D Action Recognition (2017, methodology): “For all datasets, the clips are generated with all frames of the original skeleton sequence without any pre-processing such as normalization, temporal down-sampling or noise ﬁl-tering.”
Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks (2017, background): “Recently, a primal-dual framework for real-time dense RGB-D scene ﬂow [23] has been proposed.”
Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition (2016, methodology): “It is worth noting that deep LSTM [51], Co-occurrence LSTM [51], and HBRNN [30] all use the Svaitzky-Golay ﬁlter in temporal domain to smooth the skeleton joint positions to reduce the inﬂuence of the noise in the data captured by Kinect.”
An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data (2016, methodology): “However, in comparisons to the holistic based approaches (Simonyan and Zisserman 2014; Wu et al. 2015; Zhu et al. 2016) which use all the frames, it loses some information.”
Action Recognition Based on Joint Trajectory Maps Using Convolutional Neural Networks (2016, background): “Recently, Recurrent Neural Networks (RNNs) [3; 15; 28; 13] have also been adopted for action recognition from skeleton data.”
Online Human Action Detection using Joint Classification-Regression Recurrent Neural Networks (2016, methodology): “This is the state-of-the-art skeleton based action recognition method Deep LSTM [15], which employs a deep LSTM network (with three LSTM layers and three fully connected layers) that achieves good results on many action recognition datasets.”