Skeleton-Based Action Recognition With Multi-Stream Adaptive Graph Convolutional Networks

By Yifan Zhang, Jian Cheng, Hanqing Lu. IEEE Transactions on Image Processing, 2019

543 citations recorded by the source corpus.

Graph convolutional networks (GCNs), which generalize CNNs to more generic non-Euclidean structures, have achieved remarkable performance for skeleton-based action recognition. However, there still exist several issues in the previous GCN-based models. First, the topology of the graph is set heuristically and fixed over all the model layers and input data. This may not be suitable for the hierarchy of the GCN model and the diversity of the data in action recognition tasks. Second, the second-order information of the skeleton data, i.e., the length and orientation of the bones, is rarely investigated, which is naturally more informative and discriminative for the human action recognition. In this work, we propose a novel multi-stream attention-enhanced adaptive graph convolutional neural network (MS-AAGCN) for skeleton-based action recognition. The graph topology in our model can be either uniformly or individually learned based on the input data in an end-to-end manner. This data-driven approach increases the flexibility of the model for graph construction and brings more generality to adapt to various data samples. Besides, the proposed adaptive graph convolutional layer is further enhanced by a spatial-temporal-channel attention module, which helps the model pay more attention to important joints, frames and features. Moreover, the information of both the joints and bones, together with their motion information, are simultaneously modeled in a multi-stream framework, which shows notable improvement for the recognition accuracy. Extensive experiments on the two large-scale datasets, NTU-RGBD and Kinetics-Skeleton, demonstrate that the performance of our model exceeds the state-of-the-art with a significant margin.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c1c9e6aa53af2a72a12a68ffccc1973f09792f45

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Liang Wang
2. Hong Liu
3. Zhang Zhang
4. Caifeng Shan
5. Yisheng Song
6. Bing Yang
7. Sicheng Li
8. Qinghan Li

In Biomedical Engineering

1. Zhang Zhang
2. Caifeng Shan
3. Yisheng Song
4. Liang Wang
5. Hong Liu
6. Qiaoyong Zhong
7. Shiliang Pu
8. Di Xie

In Computer Vision and Pattern Recognition

1. Liang Wang
2. Zhang Zhang
3. Caifeng Shan
4. Yisheng Song
5. Hong Liu
6. Shiliang Pu
7. Di Xie
8. Qiaoyong Zhong

Citations in the corpus

• Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition (2021, methodology): “ST-GCN (Yan, Xiong, and Lin 2018) 31.6 53.7 AS-GCN (Li et al. 2019a) 34.8 56.3 2s AGCN (Shi et al. 2019c) 36.1 58.7 4s DGNN (Shi et al. 2019a) 36.9 56.9 2s NAS-GCN (Peng et al. 2020) 37.1 60.1 2s MS-G3D (Liu et al. 2020) 38.0 60.9 Js”
• Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition (2020, methodology): “In the previous GCN-based methods [25, 26, 33], the number of nodes in the graph is kept unchanged.”
• Constructing Stronger and Faster Baselines for Skeleton-Based Action Recognition (2021, methodology): “Previous attention modules for skeleton-based action recognition are mainly implemented by a Multi-layer Perception (MLP) like SENet structure [37], such as AGC-LSTM [41] and MS-AAGCN [26].”