Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning

By Chenyang Si, Ya Jing, Wei Wang, Liang Wang, Tien-Ping Tan. European Conference on Computer Vision, 2018

376 citations recorded by the source corpus.

Skeleton-based action recognition has made great progress recently, but many problems still remain unsolved. For example, the representations of skeleton sequences captured by most of the previous methods lack spatial structure information and detailed temporal dynamics features. In this paper, we propose a novel model with spatial reasoning and temporal stack learning (SR-TSL) for skeleton-based action recognition, which consists of a spatial reasoning network (SRN) and a temporal stack learning network (TSLN). The SRN can capture the high-level spatial structural information within each frame by a residual graph neural network, while the TSLN can model the detailed temporal dynamics of skeleton sequences by a composition of multiple skip-clip LSTMs. During training, we propose a clip-based incremental loss to optimize the model. We perform extensive experiments on the SYSU 3D Human-Object Interaction dataset and NTU RGB+D dataset and verify the effectiveness of each network of our model. The comparison results illustrate that our approach achieves much better results than the state-of-the-art methods.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a84a7885af7b8c310d9c6b25c52420553bd8ef30

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Liang Wang
2. Hanqing Lu
3. Yifan Zhang
4. Jian Cheng
5. Jun Liu
6. Zhang Zhang
7. Caifeng Shan
8. Yisheng Song

In Computer Vision and Pattern Recognition

1. Liang Wang
2. Jun Liu
3. Hanqing Lu
4. Hossein Rahmani
5. G. Wang
6. Jian Cheng
7. Qiuhong Ke
8. Yi Yang

Citations in the corpus

• Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching (2019, methodology): “Deep STGCK (Li et al. 2018) 74.9 86.3 AAAI2018 ST-GCN (Yan et al. 2018) 81.5 88.3 AAAI2018 DPRL (Tang et al. 2018) 83.5 89.8 CVPR2018 SR-TSL (Si et al. 2018) 84.8 92.4 ECCV2018 STGR-GCN (Li et al. 2019a) 86.9 92.3 AAAI2019 GR-GCN (Gao et al. 2019) 87.5 94.3 ACMM2019 AS-GCN (Li et al. 2019b) 86 (Li…”
• Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition (2020, methodology): “Data preprocessing is essential for skeletonbased action recognition, according to the previous studies [23, 25, 27].”
• Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition (2020, background): “For this reason, recent approaches [50, 19, 34, 35, 32] model the joint movement patterns of an action with a skeleton spatial-temporal graph, which is a series of disjoint and isomorphic skeleton graphs at different time steps carrying information in both spatial and temporal dimensions.”
• Constructing Stronger and Faster Baselines for Skeleton-Based Action Recognition (2021, methodology): “, ST-GCN [11], SR-TSL [42], AS-GCN [48] and 2s-AGCN [13], are implemented by ourselves, based on their released codes.”
• Actional-Structural Graph Convolutional Networks for Skeleton-Based Action Recognition (2019, methodology): “Specifically, STGCN [29] combines GCN with temporal CNN to capture spatio-temporal features, and SR-TSL [23] use gated recurrent unit (GRU) to propagate messages on graphs and use LSTM to learn the temporal features.”
• An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition (2019, methodology): “[21] apply graph neural networks to capture spatial structural information and then use LSTM to model temporal dynamics.”
• Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition (2019, methodology): “Our proposed SGN is still superior to [48] and [33] by 5.”
• Dynamic Multiscale Graph Neural Networks for 3D Skeleton Based Human Motion Prediction (2020, background): “on graph representation learning and the relative applications [29, 8, 22, 16, 46, 35].”
• Skeleton-Based Action Recognition With Shift Graph Convolutional Network (2020, background): “In the computer vision field, skeleton-based human action recognition has attracted much attention due to its robustness against the dynamic circumstance and complicated background [2,12,14,16,20,21,23–25,27,30,34,36].”
• Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “As a result, many GNN and GCN-based HAR methods [152], [186] have been proposed to treat the skele-”