Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition

By Jun Liu, Amir Shahroudy, Dong Xu, Gang Wang. 2016

1,227 citations recorded by the source corpus.

3D action recognition – analysis of human actions based on 3D skeleton data – becomes popular recently due to its succinctness, robustness, and view-invariant representation. Recent attempts on this problem suggested to develop RNN-based learning methods to model the contextual dependency in the temporal domain. In this paper, we extend this idea to spatio-temporal domains to analyze the hidden sources of action-related information within the input data over both domains concurrently. Inspired by the graphical structure of the human skeleton, we further propose a more powerful tree-structure based traversal method. To handle the noise and occlusion in 3D skeleton data, we introduce new gating mechanism within LSTM to learn the reliability of the sequential input data and accordingly adjust its effect on updating the long-term context information stored in the memory cell. Our method achieves state-of-the-art performance on 4 challenging benchmark datasets for 3D human action analysis.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Jun Liu
2. Hanqing Lu
3. Yifan Zhang
4. Jian Cheng
5. G. Wang
6. Hossein Rahmani
7. Liang Wang
8. Wanqing Li

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. G. Wang
3. Xiaogang Wang
4. Dacheng Tao
5. Yi Yang
6. Jun Liu
7. Ming-Hsuan Yang
8. Trevor Darrell

In Model Architectures

1. G. Wang
2. Jun Liu
3. Hanqing Lu
4. Yifan Zhang
5. Jian Cheng
6. Alex ChiChung Kot
7. Ling-yu Duan
8. Amir Shahroudy

Citations in the corpus

• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, methodology): “Liu et al. [48] introduced a Spatio-Temporal LSTM network by modeling the context information in both temporal and spatial dimensions.”
• Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos (2016, background): “[33, 32] extended the recurrent network based sequence analysis towards sequences of body joints.”
• Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, background): “research domain [26], [27], and a series of hand-crafted features [28], [29], [30], [31], [32], [33], [34], [35], [36], [37] and deep learning-based approaches [18], [19], [38], [39], [40], [41], [42], [43], [44] have been proposed.”
• Feature Boosting Network For 3D Pose Estimation (2019, methodology): “To the best of our knowledge, the proposed work is the first of its nature in introducing gating mechanisms [28] in a CNN architecture for modulating and propagating the features by considering their context consistency for 3D pose estimation.”
• 2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning (2018, methodology): “To cope with noisy skeletons, Spatio-Temporal LSTM networks have been widely used by applying a gating mechanism [26] to learn the reliability of skeleton sequences or by using attention mechanisms [27, 46].”
• Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “The usage of RGB-D cameras is very popular for this case [44, 34, 56], as it allows for the use of articulated pose (skeletons) to be delivered in real time and relatively cheaply by some middleware.”
• RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “To exploit this dependency, Liu et al.[131] proposed a spatio-temporal LSTM (ST-LSTM) network which extends the traditional LSTM-based 22 learning to both temporal and spatial domains.”
• When Kernel Methods Meet Feature Learning: Log-Covariance Network for Action Recognition From Skeletal Data (2017, methodology): “If compared to [32, 21, 15, 5], the training time in this case is much longer even if using GPU acceleration.”
• Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks (2017, methodology): “Readers are referred to [29] for more details about the mechanism of ST-LSTM.”
• Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates (2017, methodology): “In [62], we validated the effectiveness of our model on four benchmark datasets.”
• Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks (2017, methodology): “[30] extend LSTM to spatial-temporal domain for the purpose of modeling the dependencies between joints.”
• Interpretable 3D Human Action Analysis with Temporal Convolutional Networks (2017, methodology): “Deep spatio-temporal LSTM with Trust Gates is introduced in [23] to learn features both in the temporal and spatial domains.”
• Pose-conditioned Spatio-Temporal Attention for Human Action Recognition (2017, methodology): “In the lines of [23], we define a topological ordering of the joints in a human body as a connected cyclic path over joints (see figure 2a).”
• Learning and Refining of Privileged Information-Based RNNs for Action Recognition from Depth Sequences (2017, background): “RNN-based approaches become the dominant solution [61, 42, 9, 27] for action recognition from depth sequence recently.”
• View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data (2017, background): “On one hand, it loses partial motion information, e.g ., the moving trajectory and speed of the body center, and the changing dynamics of the body orientation.”
• A New Representation of Skeleton Sequences for 3D Action Recognition (2017, methodology): “All of the mentioned works directly operate on the native 3D coordinates of the joints to extract features and learn models.”
• Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks (2017, background): “However, using RNNs did not give an improvement over temporal pooling of convolutional features [82] or even over hand-crafted features.”
• Deep Learning on Lie Groups for Skeleton-Based Action Recognition (2016, background): “Due to the development of depth sensors, 3D human activity analysis [27, 45, 23, 43, 41, 3, 42, 37, 44, 26, 35, 17] has attracted more interest than ever before.”
• Action Recognition Based on Joint Trajectory Maps Using Convolutional Neural Networks (2016, neutral): “ew Lie Group [34] 50.08% 52.76% Dynamic Skeletons [38] 60.23% 65.22% HBRNN [43] 59.07% 63.97% 2 Layer RNN [46] 56.29% 64.09% 2 Layer LSTM [46] 60.69% 67.29% Part-aware LSTM [46] 62.93% 70.27% ST-LSTM [47] 65.20% 76.10% ST-LSTM+ Trust Gate [47] 69.20% 77.70% Proposed Method 76.32% 81.08% method achie”