Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition

By Ken Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, Wanli Ouyang. Computer Vision and Pattern Recognition, 2020

1,080 citations recorded by the source corpus.

Spatial-temporal graphs have been widely used by skeleton-based action recognition algorithms to model human action dynamics. To capture robust movement patterns from these graphs, long-range and multi-scale context aggregation and spatial-temporal dependency modeling are critical aspects of a powerful feature extractor. However, existing methods have limitations in achieving (1) unbiased long-range joint relationship modeling under multi-scale operators and (2) unobstructed cross-spacetime information flow for capturing complex spatial-temporal dependencies. In this work, we present (1) a simple method to disentangle multi-scale graph convolutions and (2) a unified spatial-temporal graph convolutional operator named G3D. The proposed multi-scale aggregation scheme disentangles the importance of nodes in different neighborhoods for effective long-range modeling. The proposed G3D module leverages dense cross-spacetime edges as skip connections for direct information propagation across the spatial-temporal graph. By coupling these proposals, we develop a powerful feature extractor named MS-G3D based on which our model outperforms previous state-of-the-art methods on three large-scale datasets: NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3950b578a93674d7a8dc6829581058f76823136e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities (2022, methodology): “Implementation details: We use the publicly available PyTorch [23] code for 2s-AGCN and MS-G3D.”
Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition (2020, methodology): “1 MS-G3D Net (2 pathways) [21] ~8.”
Revisiting Skeleton-based Action Recognition (2021, methodology): “Upon the baseline, adjacency powering is used for multiscale modeling [22, 19] while self-attention mechanisms are utilized to improve the modeling capacity [27, 18].”
Constructing Stronger and Faster Baselines for Skeleton-Based Action Recognition (2021, methodology): “The inferior performance of the EfficientGCN in cross-view action recognition can be explained from two aspects: 1) in contrast to the 3-stream fusion in the EfficientGCN, the DC-GCN+ADG, the 4sShift-GCN and the MST-GCN all adopt 4-stream (joint, bone, motion, and bone motion) fusion strategy, where”
MotionBERT: A Unified Perspective on Learning Human Motion Representations (2022, methodology): “Towards modeling the spatio-temporal relationship among human joints, previous studies mainly employ LSTM [98, 138] and GCN [21, 55, 68, 96, 123].”
Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition (2021, methodology): “Most GCN-based methods follow topology-shared manner, including aforementioned static methods [9, 20, 31] and dynamic methods [15, 24, 33, 34].”
Human Action Recognition From Various Data Modalities: A Review (2020, neutral): “[199] integrated a disentangled multi-scale aggregation”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, background): “Recently, there has also been an increase in interest for explicitly reasoning about human-object interactions [16, 18, 28, 31, 39, 50, 63, 81, 86, 91, 97] and skeletal action recognition [10, 45, 52, 66, 92], however mostly from third-person viewpoints.”
Else-Net: Elastic Semantic Network for Continual Action Recognition from Skeleton Data (2021, methodology): “Besides, we conduct an experiment on training our back-bone encoder, i.e., MS-G3D [20] in the continual learning manner.”
InfoGCN: Representation Learning for Human Skeleton-based Action Recognition (2022, methodology): “…[44] 88.5 95.1 SGN [60] 89.0 94.5 DGNN [43] 89.9 96.1 ST-TR-agcn [39] 90.3 96.3 Shift-GCN [9] 90.7 96.5 DC-GCN+ADG [8] 90.8 96.6 PA-ResGCN-B19 [47] 90.9 96.0 DDGCN [22] 91.1 97.1 Dynamic GCN [58] 91.5 96.0 MS-G3D [33] 91.5 96.2 MST-GCN [7] 91.5 96.6 CTR-GCN [6] 92.4 96.8 Ours 93.0 97.1 Acc (”
Skeleton-based action recognition via spatial and temporal transformer networks (2020, background): “Despite being proven to perform very well on skeleton data, ST-GCN models have some structural limitations, some of them already addressed by [41, 44, 6, 32].”
Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition (2021, methodology): “ST-GCN (Yan, Xiong, and Lin 2018) 31.6 53.7 AS-GCN (Li et al. 2019a) 34.8 56.3 2s AGCN (Shi et al. 2019c) 36.1 58.7 4s DGNN (Shi et al. 2019a) 36.9 56.9 2s NAS-GCN (Peng et al. 2020) 37.1 60.1 2s MS-G3D (Liu et al. 2020) 38.0 60.9 Js”
Spatial Temporal Transformer Network for Skeleton-based Action Recognition (2020, background): “However, ST-GCN models have some structural limitations, some of them already addressed in [13,14,2,10]: (i) The topology of the graph representing the human body is fixed for all layers and all the actions, preventing the extraction of rich representations.”