Skeleton-Based Action Recognition With Directed Graph Neural Networks

By Lei Shi, Yifan Zhang, Jian Cheng, Hanqing Lu. Computer Vision and Pattern Recognition, 2019

798 citations recorded by the source corpus.

The skeleton data have been widely used for the action recognition tasks since they can robustly accommodate dynamic circumstances and complex backgrounds. In existing methods, both the joint and bone information in skeleton data have been proved to be of great help for action recognition tasks. However, how to incorporate these two types of data to best take advantage of the relationship between joints and bones remains a problem to be solved. In this work, we represent the skeleton data as a directed acyclic graph based on the kinematic dependency between the joints and bones in the natural human body. A novel directed graph neural network is designed specially to extract the information of joints, bones and their relations and make prediction based on the extracted features. In addition, to better fit the action recognition task, the topological structure of the graph is made adaptive based on the training process, which brings notable improvement. Moreover, the motion information of the skeleton sequence is exploited and combined with the spatial information to further enhance the performance in a two-stream framework. Our final model is tested on two large-scale datasets, NTU-RGBD and Skeleton-Kinetics, and exceeds state-of-the-art performance on both of them.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/68a024d7b70ef3989a6751678f635cbe754440fc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Jun Liu
2. Hanqing Lu
3. Yifan Zhang
4. Jian Cheng
5. Liang Wang
6. Hossein Rahmani
7. Qiuhong Ke
8. Zhang Zhang

In Computer Vision and Pattern Recognition

1. Jun Liu
2. Liang Wang
3. Hossein Rahmani
4. Hanqing Lu
5. Jian Cheng
6. Qiuhong Ke
7. G. Wang
8. Qi Tian

In Human-Computer Interaction

1. Liang Wang
2. Yifan Zhang
3. Jian Cheng
4. Hanqing Lu
5. Jun Liu
6. Hong Liu
7. Lei Shi
8. Caifeng Shan

Citations in the corpus

• Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition (2020, methodology): “4.6† 75.3† GR-GCN [7] ACMMM19 – – 87.5 94.3 – – AS-GCN [14] CVPR19 – 6.99★ 86.8 94.2 77.9† 78.5† 2s-AGCN [23] CVPR19 22.3† 6.94★ 88.5 95.1 82.5† 84.2† AGC-LSTM [24] CVPR19 – 22.89★ 89.2 95.0 – – DGNN [22] CVPR19 – 26.24★ 89.9 96.1 – – AS-GCN+DH-TCN [18] arXiv19 – – 85.3 92.8 78.3 79.8 SGN [35] arXiv”
• Skeleton-Based Action Recognition With Shift Graph Convolutional Network (2020, methodology): “Some recent works even reach ∼100 GFLOPs [20] due to introducing incremental modules and multi-stream fusion strategy.”
• Skeleton-based action recognition via spatial and temporal transformer networks (2020, methodology): “The sub-networks outputs are eventually fused together by summing up their softmax output scores to obtain the final prediction, as proposed by Shi et al. (2019b) and Shi et al. (2019a).”
• Decoupling GCN with DropGraph Module for Skeleton-Based Action Recognition (2020, methodology): “GCN-based action recognition models [46,44,34,33,13,35,18] are composed of several spatial-temporal GCN blocks, where spatial graph convolution is the key component.”
• Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition (2020, neutral): “0 2019 DGNN [24] 89.”
• Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “[187] represented the skeleton as a directed acyclic graph to effectively incorporate both the joint and bone information, and utilized a GNN to perform HAR.”
• InfoGCN: Representation Learning for Human Skeleton-based Action Recognition (2022, methodology): “Since the introduction of GCN [21], various approaches exploiting the graph structure of extrinsic topology have been introduced [26, 33, 43, 56].”
• Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition (2021, methodology): “ST-GCN (Yan, Xiong, and Lin 2018) 31.6 53.7 AS-GCN (Li et al. 2019a) 34.8 56.3 2s AGCN (Shi et al. 2019c) 36.1 58.7 4s DGNN (Shi et al. 2019a) 36.9 56.9 2s NAS-GCN (Peng et al. 2020) 37.1 60.1 2s MS-G3D (Liu et al. 2020) 38.0 60.9 Js”
• Constructing Stronger and Faster Baselines for Skeleton-Based Action Recognition (2021, background): “And the DGNN [16] contains more than 26 million parameters, which makes it very hard to do parameter tuning on large-scale datasets.”
• Dynamic Multiscale Graph Neural Networks for 3D Skeleton Based Human Motion Prediction (2020, methodology): “Several graph-based models have been employed for skeleton-based action recognition [46, 27, 34], motion prediction [31] and 3D pose estimation [47]; Different from any previous works, our model considers multiscale graphs and corresponding operations.”
• Is Homophily a Necessity for Graph Neural Networks? (2021, background): “They have been widely adopted to tackle numerous applications from various domains [18, 8, 2, 31].”
• Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition (2020, methodology): “In particular, skeleton-based human action recognition involves predicting actions from skeleton representations of human bodies instead of raw RGB videos, and the significant results seen in recent work [50, 33, 32, 34, 21, 20, 54, 35] have proven its merits.”