Skeleton-Based Action Recognition With Shift Graph Convolutional Network

By Ke Cheng, Yifan Zhang, Xiangyu He, Weihan Chen, Jian Cheng, Hanqing Lu. Computer Vision and Pattern Recognition, 2020

845 citations recorded by the source corpus.

Action recognition with skeleton data is attracting more attention in computer vision. Recently, graph convolutional networks (GCNs), which model the human body skeletons as spatiotemporal graphs, have obtained remarkable performance. However, the computational complexity of GCN-based methods are pretty heavy, typically over 15 GFLOPs for one action sample. Recent works even reach about 100 GFLOPs. Another shortcoming is that the receptive fields of both spatial graph and temporal graph are inflexible. Although some works enhance the expressiveness of spatial graph by introducing incremental adaptive modules, their performance is still limited by regular GCN structures. In this paper, we propose a novel shift graph convolutional network (Shift-GCN) to overcome both shortcomings. Instead of using heavy regular graph convolutions, our Shift-GCN is composed of novel shift graph operations and lightweight point-wise convolutions, where the shift graph operations provide flexible receptive fields for both spatial graph and temporal graph. On three datasets for skeleton-based action recognition, the proposed Shift-GCN notably exceeds the state-of-the-art methods with more than 10 times less computational complexity.

Subfields: Computer Vision and Pattern Recognition, Rehabilitation

View Paper: https://www.semanticscholar.org/paper/1d41090ba8e597380e7353b6b1f42d6a7d9f83b4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Human Action Recognition From Various Data Modalities: A Review (2020, background): “[203], which adopts shift graph operations and lightweight point-wise convolutions, instead of using heavy regular graph convolutions.”
MotionBERT: A Unified Perspective on Learning Human Motion Representations (2022, methodology): “To-wards modeling the spatio-temporal relationship among human joints, previous studies mainly employ LSTM [91,129] and GCN [20, 51, 64, 89, 116].”
Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition (2021, methodology): “To conduct a fair comparison, we adopt the same multi-stream fusion strategy as Shi et al. (2019a,b); Cheng et al. (2020), which utilizes the joint, bone, joint motion and bone motion streams.”
InfoGCN: Representation Learning for Human Skeleton-based Action Recognition (2022, methodology): “The 4-stream ensemble [6, 7, 9, 43] is adopted for the baseline to compare with the ensemble results of models trained with multi-modal representations as inputs.”
Spatial Temporal Transformer Network for Skeleton-based Action Recognition (2020, background): “However, ST-GCN models have some structural limitations, some of them already addressed in [13,14,2,10]: (i) The topology of the graph representing the human body is fixed for all layers and all the actions, preventing the extraction of rich representations.”
Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition (2021, methodology): “We set the neighborhood of each joint as the entire human skeleton graph, which is proved to be more effective in this task by previous work [4, 24].”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, background): “Recently, there has also been an increase in interest for explicitly reasoning about human-object interactions [16, 18, 28, 31, 39, 50, 63, 81, 86, 91, 97] and skeletal action recognition [10, 45, 52, 66, 92], however mostly from third-person viewpoints.”
A Survey of Human-in-the-loop for Machine Learning (2021, methodology): “In recent years, neural network-based Deep Learning methods (DL) have emerged as the state-of-the-art technique for performing many computer vision tasks [108, 109, 110, 111].”