Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks

By Jun Liu, G. Wang, Ling-yu Duan, Kamila Abdiyeva, Alex ChiChung Kot. 2017

539 citations recorded by the source corpus.

Human action recognition in 3D skeleton sequences has attracted a lot of research attention. Recently, long short-term memory (LSTM) networks have shown promising performance in this task due to their strengths in modeling the dependencies and dynamics in sequential data. As not all skeletal joints are informative for action recognition, and the irrelevant joints often bring noise which can degrade the performance, we need to pay more attention to the informative ones. However, the original LSTM network does not have explicit attention ability. In this paper, we propose a new class of LSTM network, global context-aware attention LSTM, for skeleton-based action recognition, which is capable of selectively focusing on the informative joints in each frame by using a global context memory cell. To further improve the attention capability, we also introduce a recurrent attention mechanism, with which the attention performance of our network can be enhanced progressively. Besides, a two-stream framework, which leverages coarse-grained attention and fine-grained attention, is also introduced. The proposed method achieves state-of-the-art performance on five challenging datasets for skeleton-based action recognition.

Subfields: Computer Vision and Pattern Recognition, Model Architectures, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

In Human-Computer Interaction

Citations in the corpus

NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, methodology): “We take six methods [30], [47], [48], [94], [103], [105] as examples, and show their results achieved by using different sizes of training set in Fig.”
Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, background): “As a result, activity analysis with 3D skeleton data becomes a popular domain of research [9–17] thanks to its succinctness, high level representation, and robustness against variations in viewpoints, illumination, clothing textures, and background clutter [1, 18, 19].”
Learning Regularity in Skeleton Trajectories for Anomaly Detection in Videos (2019, background): “ion in video scenes is an important factor for studying social behavior. It has been applied in multiple computer vision applications, mostly with supervised learning tasks such as action recognition [8, 19] and person re-identiﬁcation [9]. Recently, more effort has been invested into unsupervised l”
Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition (2020, unknown): “ettings in terms of top-1 accuracy (%). Method C-Subject C-Setup Year Dynamic Skeleton [7] 50.8 54.7 2015 PA-LSTM [23] 25.5 26.3 2016 ST-LSTM+TS [18] 55.7 57.9 2016 FSNet [17] 59.9 62.4 2019 2s-ALSTM [19] 61.2 63.3 2017 MT CNN [9] 62.2 61.8 2018 BPEM [20] 64.6 66.9 2018 2s-AGCN [26] 82.9 84.9 2019 M”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “Hence, various methods [155], [156], [157], [158] have applied and adapted RNNs and LSTMs to effectively model the temporal context information within the skeleton sequences for HAR.”
Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition (2020, background): “Table 4 compares non-graph [26, 27, 16, 28] and graph-based methods [33].”
Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition (2019, unknown): “N - 86.6 93.4 Table 5: Performance comparisons on the SYSU dataset in terms of accuracy (%). Method Year CS SS VA-LSTM [51] 2017 77.5 76.9 ST-LSTM [21] 2018 76.5 - DPRL+GCNN [37] 2018 76.9 - GCA-LSTM [23] 2018 78.6 - SR-TSL [33] 2018 81.9 80.7 ElAtt-GRU [53] 2018 85.7 85.7 SGN w/o Sem. - 84.4 83.1 S”
Skeleton-based action recognition via spatial and temporal transformer networks (2020, unknown): “9 GCA-LSTM [28] 61.”
SSNet: Scale Selection Network for Online 3D Action Prediction (2018, background): “As a result, activity analysis with 3D skeleton data becomes popular [19, 49, 62, 41, 81, 57, 43] due to its succinctness, high level representation, and robustness against variations in viewpoints, clothing textures, illumination, and background [11, 42, 23].”