Actional-Structural Graph Convolutional Networks for Skeleton-Based Action Recognition

By Maosen Li, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang, Qi Tian. Computer Vision and Pattern Recognition, 2019

1,063 citations recorded by the source corpus.

Action recognition with skeleton data has recently attracted much attention in computer vision. Previous studies are mostly based on fixed skeleton graphs, only capturing local physical dependencies among joints, which may miss implicit joint correlations. To capture richer dependencies, we introduce an encoder-decoder structure, called A-link inference module, to capture action-specific latent dependencies, i.e. actional links, directly from actions. We also extend the existing skeleton graphs to represent higher-order dependencies, i.e. structural links. Combing the two types of links into a generalized skeleton graph, We further propose the actional-structural graph convolution network (AS-GCN), which stacks actional-structural graph convolution and temporal convolution as a basic building block, to learn both spatial and temporal features for action recognition. A future pose prediction head is added in parallel to the recognition head to help capture more detailed action patterns through self-supervision. We validate AS-GCN in action recognition using two skeleton data sets, NTU-RGB+D and Kinetics. The proposed AS-GCN achieves consistently large improvement compared to the state-of-the-art methods. As a side product, AS-GCN also shows promising results for future pose prediction.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/814b70cd133f97ef039bcc44124d9344dd8b3f64

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Liu
2. Hossein Rahmani
3. Qiuhong Ke
4. Jian Cheng
5. Hanqing Lu
6. Yifan Zhang
7. Dahua Lin
8. Zehua Sun

In Biomedical Engineering

1. Jun Liu
2. Liang Wang
3. Hanqing Lu
4. Yifan Zhang
5. Jian Cheng
6. Hossein Rahmani
7. Qiuhong Ke
8. Zhang Zhang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Xiaogang Wang
4. Yi Yang
5. Jun Liu
6. Ming-Hsuan Yang
7. Wanli Ouyang
8. Jitendra Malik

Citations in the corpus

• Skeleton-Based Action Recognition With Shift Graph Convolutional Network (2020, methodology): “Many variants of ST-GCN are proposed [9,12,20–22,31], which typically introduce incremental modules to enhance the expressiveness and network capacity.”
• Revisiting Skeleton-based Action Recognition (2021, methodology): “Upon the baseline, adjacency powering is used for multiscale modeling [22, 19] while self-attention mechanisms are utilized to improve the modeling capacity [27, 18].”
• Decoupling GCN with DropGraph Module for Skeleton-Based Action Recognition (2020, methodology): “GCN-based action recognition models [46,44,34,33,13,35,18] are composed of several spatial-temporal GCN blocks, where spatial graph convolution is the key component.”
• Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition (2021, methodology): “Most GCN-based methods follow topology-shared manner, including aforementioned static methods [9, 20, 31] and dynamic methods [15, 24, 33, 34].”
• Iterative Context-Aware Graph Inference for Visual Dialog (2020, background): “Graph neural networks have attracted attention in various tasks [32, 22, 17, 8, 26].”
• Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition (2020, methodology): “, ST-GCN [31], RA-GCN [27], ASGCN [14] and 2s-AGCN [23], are implemented by ourselves, based on their released codes.”
• InfoGCN: Representation Learning for Human Skeleton-based Action Recognition (2022, methodology): “Graph representation of the skeleton using bone connectiv-20186 ity (extrinsic topology) [27, 33, 44, 56, 60] has an inherent limitation: it can ignore possible joint relations, called an intrinsic topology.”
• Dynamic Multiscale Graph Neural Networks for 3D Skeleton Based Human Motion Prediction (2020, methodology): “Several graph-based models have been employed for skeleton-based action recognition [46, 27, 34], motion prediction [31] and 3D pose estimation [47]; Different from any previous works, our model considers multiscale graphs and corresponding operations.”
• Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition (2019, neutral): “Biologically, human is able to recognize the action category by observing only the motion VA-LSTM[57] HCN[23] ST-GCN[54] 2s-AGCN[37] AS-GCN[24] SR-TSL[40] AGC-LSTM(joint)[39] VA-CNN[58] SGN(ours) of joints even without appearance information [17].”
• MotionBERT: A Unified Perspective on Learning Human Motion Representations (2022, methodology): “To-wards modeling the spatio-temporal relationship among human joints, previous studies mainly employ LSTM [91,129] and GCN [20, 51, 64, 89, 116].”
• Human Action Recognition From Various Data Modalities: A Review (2020, background): “[193] further proposed an Actional-Structural GCN (ASGCN), combining actional links and structural links into a generalized skeleton graph.”
• Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching (2019, methodology): “Currently, Graph Convolutional Networks (GCN) (Kipf and Welling 2016; Defferrard, Bresson, and Vandergheynst 2016) has been introduced to skeleton-based action recognition and achieved many encouraging results (Li et al. 2018; Yan, Xiong, and Lin 2018; Li et al. 2019a; Gao et al. 2019; Shi et al. 20”
• H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, background): “Recently, there has also been an increase in interest for explicitly reasoning about human-object interactions [16, 18, 28, 31, 39, 50, 63, 81, 86, 91, 97] and skeletal action recognition [10, 45, 52, 66, 92], however mostly from third-person viewpoints.”
• Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition (2021, background): “ST-GCN (Yan, Xiong, and Lin 2018) 31.6 53.7 AS-GCN (Li et al. 2019a) 34.8 56.3 2s AGCN (Shi et al. 2019c) 36.1 58.7 4s DGNN (Shi et al. 2019a) 36.9 56.9 2s NAS-GCN (Peng et al. 2020) 37.1 60.1 2s MS-G3D (Liu et al. 2020) 38.0 60.9 Js”
• Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition (2020, methodology): “In particular, skeleton-based human action recognition involves predicting actions from skeleton representations of human bodies instead of raw RGB videos, and the significant results seen in recent work [50, 33, 32, 34, 21, 20, 54, 35] have proven its merits.”
• Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition (2020, neutral): “oth top-1 and top-5 accuracies (%) are reported. Method Top-1 Acc. Top-5 Acc. Year Deep LSTM [23] 16.4 35.3 2016 TCN [10] 20.3 40.0 2017 ST-GCN [33] 30.7 52.8 2018 STGR-GCN [12] 33.6 56.1 2019 AS-GCN [15] 34.8 56.5 2019 2s-AGCN [26] 36.1 58.7 2019 MS-AAGCN [25] 37.8 61.0 2020 MS-G3D Net [21] 38.0 60”