Online Human Action Detection using Joint Classification-Regression Recurrent Neural Networks

By Yanghao Li, Cuiling Lan, Junliang Xing, Wenjun Zeng, Chunfen Yuan, Jiaying Liu. 2016

225 citations recorded by the source corpus.

Human action recognition from well-segmented 3D skeleton data has been intensively studied and has been attracting an increasing attention. Online action detection goes one step further and is more challenging, which identifies the action type and localizes the action positions on the fly from the untrimmed stream data. In this paper, we study the problem of online action detection from streaming skeleton data. We propose a multi-task end-to-end Joint Classification-Regression Recurrent Neural Network to better explore the action type and temporal localization information. By employing a joint classification and regression optimization objective, this network is capable of automatically localizing the start and end points of actions more accurately. Specifically, by leveraging the merits of the deep Long Short-Term Memory (LSTM) subnetwork, the proposed model automatically captures the complex long-range temporal dynamics, which naturally avoids the typical sliding window design and thus ensures high computational efficiency. Furthermore, the subtask of regression optimization provides the ability to forecast the action prior to its occurrence. To evaluate our proposed model, we build a large streaming video dataset with annotations. Experimental results on our dataset and the public G3D dataset both demonstrate very promising performance of our scheme.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, methodology): “As the action detection method, JCR-RNN [67], also estimates the start point, we also compare our method with it.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “For RNN-based methods [93, 78], even though promising results have been achieved, it is still far from satisfactory in terms of performance.”
PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding (2017, methodology): “[18] introduced a Joint Classification Regression RNN to avoid sliding window design which demonstrates state-of-the-art performance for online action detection.”
Skeleton boxes: Solving skeleton based action detection with a single deep convolutional neural network (2017, methodology): “Compared with [11], we adopt totally different framework and our offline performance is better than theirs.”
Global Context-Aware Attention LSTM Networks for 3D Action Recognition (2017, methodology): “Besides 3D action recognition, RNN and LSTM have also been applied to 3D action detection [25, 18] and forecasting [18].”
Real-Time Online Action Detection Forests Using Spatio-Temporal Contexts (2016, methodology): “OAD is a challenging problem [11, 25] and even though skeleton-based approaches [1, 52, 29, 35, 26, 25] are promising, they are bounded by the time constraint that prevents combining multiple feature modalities or considering more temporal relationships.”
Skeleton-based action recognition with convolutional neural networks (2017, methodology): “[12] Yanghao Li, Cuiling Lan, Junliang Xing, Wenjun Zeng, Chunfeng Yuan, and Jiaying Liu, "Online human action detection using joint classification-regression recurrent neural networks," in European Conference on Computer Vision.”
Spatio-Temporal Naive-Bayes Nearest-Neighbor (ST-NBNN) for Skeleton-Based Action Recognition (2017, background): “Skeleton-Based Action Recognition In recent years, skeleton-based action recognition problem attracts a lot of attention, and many learning-based methods [5, 24, 35, 21, 14, 13, 15] have been proposed.”
SSNet: Scale Selection Network for Online 3D Action Prediction (2018, methodology): “We also find our SSNet significantly outperforms the state-of-the-art RNN/LSTM based methods, JCR-RNN [38] and ST-LSTM [42] which can handle continuous skeleton sequences.”
Transition Forests: Learning Discriminative Temporal Transitions for Action Recognition and Detection (2016, methodology): “Action RF SW PCRF RNN [39] JCR-RNN [16] Ours”