An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition

By Chenyang Si, Wentao Chen, Liang Wang, Tien-Ping Tan. Computer Vision and Pattern Recognition, 2019

827 citations recorded by the source corpus.

Skeleton-based action recognition is an important task that requires the adequate understanding of movement characteristics of a human action from the given skeleton sequence. Recent studies have shown that exploring spatial and temporal features of the skeleton sequence is vital for this task. Nevertheless, how to effectively extract discriminative spatial and temporal features is still a challenging problem. In this paper, we propose a novel Attention Enhanced Graph Convolutional LSTM Network (AGC-LSTM) for human action recognition from skeleton data. The proposed AGC-LSTM can not only capture discriminative features in spatial configuration and temporal dynamics but also explore the co-occurrence relationship between spatial and temporal domains. We also present a temporal hierarchical architecture to increase temporal receptive fields of the top AGC-LSTM layer, which boosts the ability to learn the high-level semantic representation and significantly reduces the computation cost. Furthermore, to select discriminative spatial information, the attention mechanism is employed to enhance information of key joints in each AGC-LSTM layer. Experimental results on two datasets are provided: NTU RGB+D dataset and Northwestern-UCLA dataset. The comparison results demonstrate the effectiveness of our approach and show that our approach outperforms the state-of-the-art methods on both datasets.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/074611d0c9f527bc0ad06f00df779f3361e38b83

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Liu
2. Qiuhong Ke
3. Hossein Rahmani
4. Jian Cheng
5. Hanqing Lu
6. Yifan Zhang
7. Cuiling Lan
8. Wenjun Zeng

In Biomedical Engineering

1. Hanqing Lu
2. Yifan Zhang
3. Jian Cheng
4. Liang Wang
5. Jun Liu
6. Zhang Zhang
7. Caifeng Shan
8. Yisheng Song

In Computer Vision and Pattern Recognition

1. Jun Liu
2. Liang Wang
3. Hanqing Lu
4. Jian Cheng
5. Yifan Zhang
6. Hossein Rahmani
7. G. Wang
8. Qiuhong Ke

Citations in the corpus

• Skeleton-based action recognition via spatial and temporal transformer networks (2020, methodology): “As adding the di ff erential of spatial coordinates (bones information) demonstrated to lead to better results in previous works (Shi et al. (2019a); Simonyan and Zisserman (2014)), we also studied our Transformer modules on combined joint and bones information.”
• Decoupling GCN with DropGraph Module for Skeleton-Based Action Recognition (2020, methodology): “GCN-based action recognition models [46,44,34,33,13,35,18] are composed of several spatial-temporal GCN blocks, where spatial graph convolution is the key component.”
• STGAT: Modeling Spatial-Temporal Interactions for Human Trajectory Prediction (2019, background): “Recently, some meth-ods [27] [40] in the field of action recognition have made significant progress by applying GNNs to spatial-temporal”
• Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition (2020, methodology): “In particular, skeleton-based human action recognition involves predicting actions from skeleton representations of human bodies instead of raw RGB videos, and the significant results seen in recent work [50, 33, 32, 34, 21, 20, 54, 35] have proven its merits.”
• InfoGCN: Representation Learning for Human Skeleton-based Action Recognition (2022, neutral): “Lie Group [52] 74.2 Actionlet ensemble [54] 76.0 HBRNN-L [11] 78.5 Ensemble TS-LSTM [24] 89.2 AGC-LSTM [45] 93.3 Shift-GCN [9] 94.6 DC-GCN+ADG [8] 95.3 CTR-GCN [6] 96.5 Ours 97.0 Table 1.”
• Deep Learning with Graph Convolutional Networks: An Overview and Latest Applications in Computational Intelligence (2023, methodology): “After the concept of GCN was proposed, new forms of graph convolutional network models based on spectraldomain have been proposed, such as AGCN [16], AGC [17], and so on.”
• A review on the long short-term memory model (2020, background): “Naturally, there are plenty of other contributions in computer vision (Luo et al. 2018; Feng et al. 2019b; Perrett and Damen 2019; Si et al. 2019; Ma et al. 2016; Liu et al. 2017b; Guo et al. 2018; Baddar and Ro 2019), with or without proposing a variation of the LSTM model to improve performance.”
• Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “Attention mechanisms were applied for extracting discriminative information and global dependencies in [201], [202].”
• Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition (2020, background): “STA-LSTM [26] and AGC-LSTM [24] are also enhanced by attention blocks.”
• Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition (2019, background): “Previous works usually adopt heavy networks for modeling skeleton sequence of low dimensions [40, 39, 37, 58].”
• Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition (2021, background): “HBRNN (Du et al. 2015) 64.0 59.1 P-LSTM (Shahroudy et al. 2016) 67.3 60.7 TCN (Soo Kim and Reiter 2017) 83.1 74.3 VA-LSTM (Zhang et al. 2017) 87.7 79.2 ST-GCN (Yan, Xiong, and Lin 2018) 88.3 81.5 AS-GCN (Li et al. 2019a) 94.2 86.8 2s AGC-LSTM (Si et al. 2019) 95.0 89.2 2s AGCN (Shi et al. 2019c) 95.”
• Skeleton-Based Action Recognition With Shift Graph Convolutional Network (2020, methodology): “For Northwestern-UCLA, the batch size is 16, and we adopt the data pre-processing in [22].”