Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates

By Jun Liu, Amir Shahroudy, Dong Xu, Alex ChiChung Kot, G. Wang. 2017

504 citations recorded by the source corpus.

Skeleton-based human action recognition has attracted a lot of research attention during the past few years. Recent works attempted to utilize recurrent neural networks to model the temporal dependencies between the 3D positional configurations of human body joints for better analysis of human activities in the skeletal data. The proposed work extends this idea to spatial domain as well as temporal domain to better analyze the hidden sources of action-related information within the human skeleton sequences in both of these domains simultaneously. Based on the pictorial structure of Kinect's skeletal data, an effective tree-structure based traversal framework is also proposed. In order to deal with the noise in the skeletal data, a new gating mechanism within LSTM module is introduced, with which the network can learn the reliability of the sequential data and accordingly adjust the effect of the input data on the updating procedure of the long-term context representation stored in the unit's memory cell. Moreover, we introduce a novel multi-modal feature fusion strategy within the LSTM unit in this paper. The comprehensive experimental results on seven challenging benchmark datasets for human action recognition demonstrate the effectiveness of the proposed method.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, methodology): “…namely, the Part-Aware LSTM [47], the Soft RNN [102], the Dynamic Skeleton [26], the Spatio-Temporal LSTM [48], the Internal Feature Fusion [94], the GCA-LSTM [30], the Multi-Task Learning Network [49], the FSNet [103], [108], the Skeleton Visualization [104], the Two-Stream Attention LSTM…”
Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, methodology): “As a result, activity analysis with 3D skeleton data becomes a popular domain of research [9–17] thanks to its succinctness, high level representation, and robustness against variations in viewpoints, illumination, clothing textures, and background clutter [1, 18, 19].”
Video Representation Learning Using Discriminative Pooling (2018, neutral): “HMDB-51 (accuracy over 3 splits) Method Accuracy Temporal segment networks[54] 69.4% AdaScan[19] 54.9% AdaScan + IDT + C3D[19] 66.9% ST ResNet[10] 66.4% ST ResNet + IDT[10] 70.3% ST Multiplier Network[11] 68.9% ST Multiplier Network + IDT[11] 72.2% Two-stream I3D[5] 66.4% Two-stream I3D+ (Kinetics 3”
Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks (2017, methodology): “[29], [56] designed a 2D Spatio-Temporal LSTM framework to concurrently explore the hidden sources of action related context information in both temporal and spatial domains.”
Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition (2019, unknown): “R-TSL [33] 2018 84.8 92.4 SGN w/o Sem. - 84.9 90.5 SGN - 86.6 93.4 Table 5: Performance comparisons on the SYSU dataset in terms of accuracy (%). Method Year CS SS VA-LSTM [51] 2017 77.5 76.9 ST-LSTM [21] 2018 76.5 - DPRL+GCNN [37] 2018 76.9 - GCA-LSTM [23] 2018 78.6 - SR-TSL [33] 2018 81.9 80.7 ElA”
SSNet: Scale Selection Network for Online 3D Action Prediction (2018, background): “As a result, activity analysis with 3D skeleton data becomes popular [19, 49, 62, 41, 81, 57, 43] due to its succinctness, high level representation, and robustness against variations in viewpoints, clothing textures, illumination, and background [11, 42, 23].”
LSTM-CNN Architecture for Human Activity Recognition (2020, methodology): “Representative is an image processing method based on Kinect sensors which could acquire the depth image features of moving targets whereby Jun Liu et al. [10] proposed a space-time short-term memory (ST-LSTM) network to recognize activities.”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “[8] proposed a spatio-temporal LSTM network, which is able to effectively fuse the RGB and skeleton features within the LSTM unit.”