Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition

By Lei Shi, Yifan Zhang, Jian Cheng, Hanqing Lu. Computer Vision and Pattern Recognition, 2018

1,855 citations recorded by the source corpus.

In skeleton-based action recognition, graph convolutional networks (GCNs), which model the human body skeletons as spatiotemporal graphs, have achieved remarkable performance. However, in existing GCN-based methods, the topology of the graph is set manually, and it is fixed over all layers and input samples. This may not be optimal for the hierarchical GCN and diverse samples in action recognition tasks. In addition, the second-order information (the lengths and directions of bones) of the skeleton data, which is naturally more informative and discriminative for action recognition, is rarely investigated in existing methods. In this work, we propose a novel two-stream adaptive graph convolutional network (2s-AGCN) for skeleton-based action recognition. The topology of the graph in our model can be either uniformly or individually learned by the BP algorithm in an end-to-end manner. This data-driven method increases the flexibility of the model for graph construction and brings more generality to adapt to various data samples. Moreover, a two-stream framework is proposed to model both the first-order and the second-order information simultaneously, which shows notable improvement for the recognition accuracy. Extensive experiments on the two large-scale datasets, NTU-RGBD and Kinetics-Skeleton, demonstrate that the performance of our model exceeds the state-of-the-art with a significant margin.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e48f36aacb72adb74cef077c87d2351121124137

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

BABEL: Bodies, Action and Behavior with English Labels (2021, positive): “In our experiments, we use only the joint stream, which achieves 2% lower accuracy compared to 2s-AGCN [30].”
Spatial Temporal Transformer Network for Skeleton-based Action Recognition (2020, methodology): “3 1s-AGCN [14][10] 86.”
MotionBERT: A Unified Perspective on Learning Human Motion Representations (2022, methodology): “…the pretrained motion representations with a linear layer or a simple MLP. volutional Networks (ST-GCN) have been applied to modeling spatio-temporal relationship of human joints in both 3D pose estimation [12,109] and action recognition [89,116], but their connections have not been fully explored.”
Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks (2020, methodology): “ external knowledge about the attributes of each node is given, we can also set E1 = E2 = Z, where Z is a static node feature matrix. Some works have considered capturing dynamic spatial dependencies [8, 18]. In other words, they dynamically adjust the weight of two connected nodes based on temporal”
InfoGCN: Representation Learning for Human Skeleton-based Action Recognition (2022, methodology): “Shi et al. [44] introduce bone information , which is defined as a vector pointing toward its target joint from its source joint that are physically connected, as shown at k = 1 in Fig.”
Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition (2020, methodology): “2) 2s-AGCN [23] is another popular baseline in skeleton-based action recognition.”
MotionCLIP: Exposing Human Motion Generation to CLIP Space (2022, methodology): “Using MotionCLIP together with CLIP text encoder for classification yields performance marginally close to 2s-AGCN [41] dedicated architecture on the BABEL-60 benchmark.”
Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition (2020, methodology): “In particular, skeleton-based human action recognition involves predicting actions from skeleton representations of human bodies instead of raw RGB videos, and the signiﬁcant results seen in recent work [50, 33, 32, 34, 21, 20, 54, 35] have proven its merits.”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “[196] proposed a two-stream Adaptive GCN (2sAGCN), where the topology of graphs can be either uniformly or individually learned by the back-propagation”
Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities (2022, methodology): ““2s-AGCN [41]” classifies trimmed segments bounded by action start and end [ t s , t e ] .”
Decoupling GCN with DropGraph Module for Skeleton-Based Action Recognition (2020, methodology): “GCN-based action recognition models [46,44,34,33,13,35,18] are composed of several spatial-temporal GCN blocks, where spatial graph convolution is the key component.”
Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition (2020, neutral): “keleton [7] 50.8 54.7 2015 PA-LSTM [23] 25.5 26.3 2016 ST-LSTM+TS [18] 55.7 57.9 2016 FSNet [17] 59.9 62.4 2019 2s-ALSTM [19] 61.2 63.3 2017 MT CNN [9] 62.2 61.8 2018 BPEM [20] 64.6 66.9 2018 2s-AGCN [26] 82.9 84.9 2019 MS-G3D Net [21] 86.9 88.4 2020 Dynamic GCN (ours) 87.3 88.6 GCN-CNN architecture”
Revisiting Skeleton-based Action Recognition (2021, methodology): “GCN approaches for skeleton-based action recognition usually ensemble results of multiple streams (joint, bone, etc. ) to obtain better recognition performance [51].”
Constructing Stronger and Faster Baselines for Skeleton-Based Action Recognition (2021, methodology): “For example, the 2s-AGCN in [13] contains about 6.”
Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching (2019, methodology): “With a score-level fusion strategy, it gets the current best result (Shi et al. 2019).”