Spatio-Temporal Graph Convolution for Skeleton Based Action Recognition

By Chaolong Li, Zhen Cui, Wenming Zheng, Chunyan Xu, Jian Yang. 2018

187 citations recorded by the source corpus.

Variations of human body skeletons may be considered as dynamic graphs, which are generic data representation for numerous real-world applications. In this paper, we propose a spatio-temporal graph convolution (STGC) approach for assembling the successes of local convolutional filtering and sequence learning ability of autoregressive moving average. To encode dynamic graphs, the constructed multi-scale local graph convolution filters, consisting of matrices of local receptive fields and signal mappings, are recursively performed on structured graph data of temporal and spatial domain. The proposed model is generic and principled as it can be generalized into other dynamic models. We theoretically prove the stability of STGC and provide an upper-bound of the signal transformation to be learnt. Further, the proposed recursive model can be stacked into a multi-layer architecture. To evaluate our model, we conduct extensive experiments on four benchmark skeleton-based action datasets, including the large-scale challenging NTU RGB+D. The experimental results demonstrate the effectiveness of our proposed model and the improvement over the state-of-the-art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qiang Yang
2. Yu Zhang
3. Pascal Poupart
4. Rishab Goel
5. Jieping Ye
6. Ivan Kobyzev
7. Peter Forsyth
8. Kshitij Jain

In Computer Vision and Pattern Recognition

1. Qiang Yang
2. Dacheng Tao
3. Wanli Ouyang
4. Xiaogang Wang
5. Xiaopeng Hong
6. Yi Yang
7. Shuicheng Yan
8. Luc Van Gool

Citations in the corpus

• Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, neutral): “research domain [26], [27], and a series of hand-crafted features [28], [29], [30], [31], [32], [33], [34], [35], [36], [37] and deep learning-based approaches [18], [19], [38], [39], [40], [41], [42], [43], [44] have been proposed.”
• Attention Based Spatial-Temporal Graph Convolutional Networks for Traffic Flow Forecasting (2019, methodology): “Li et al. (2018) introduced graph convolutions into human action recognition tasks.”
• Representation Learning for Dynamic Graphs: A Survey (2019, methodology): “Other similar ways of embedding the M models into RNNs can be seen in Li et al. (2017b) and Seo et al. (2018). Instead of embedding a GCN into an RNN, Pareja et al. (2019) embed an RNN into a GCN by running a GCN with different parameters at each snapshot where the RNN provides the weights of the GC”
• A Survey of Deep Learning for Scientific Discovery (2020, background): “Recent work has also looked at spatio-temporal graph networks for problems like action recognition in video [124].”
• Spatiotemporal Multi-Graph Convolution Network for Ride-Hailing Demand Forecasting (2019, methodology): “ST-GCN is used in spatiotemporal prediction for skeleton based action recognition (Li et al. 2018a; Yan, Xiong, and Lin 2018).”
• Relational Representation Learning for Dynamic (Knowledge) Graphs: A Survey (2019, background): “While Li et al. [144] only use human pose features, Ghosh et al. [79] also leverage additional contextual cues such as object features, functional relationships, etc.”
• Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition (2020, methodology): “For this reason, recent approaches [50, 19, 34, 35, 32] model the joint movement patterns of an action with a skeleton spatial-temporal graph, which is a series of disjoint and isomorphic skeleton graphs at different time steps carrying information in both spatial and temporal dimensions.”
• Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching (2019, result): “Deep STGCK (Li et al. 2018) 74.9 86.3 AAAI2018 ST-GCN (Yan et al. 2018) 81.5 88.3 AAAI2018 DPRL (Tang et al. 2018) 83.5 89.8 CVPR2018 SR-TSL (Si et al. 2018) 84.8 92.4 ECCV2018 STGR-GCN (Li et al. 2019a) 86.9 92.3 AAAI2019 GR-GCN (Gao et al. 2019) 87.5 94.3 ACMM2019 AS-GCN (Li et al. 2019b) 86 (Li…”
• InfoGCN: Representation Learning for Human Skeleton-based Action Recognition (2022, methodology): “Multi-scale graph convolutions [26, 33] have been presented to capture long-range dependencies of joints.”
• Classifying Signals on Irregular Domains via Convolutional Cluster Pooling (2019, unknown): “1 Deep STGCK [17] 74.”