Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

By Sijie Yan, Yuanjun Xiong, Dahua Lin. 2018

5,308 citations recorded by the source corpus.

Dynamics of human body skeletons convey significant information for human action recognition. Conventional approaches for modeling skeletons usually rely on hand-crafted parts or traversal rules, thus resulting in limited expressive power and difficulties of generalization. In this work, we propose a novel model of dynamic skeletons called Spatial-Temporal Graph Convolutional Networks (ST-GCN), which moves beyond the limitations of previous methods by automatically learning both the spatial and temporal patterns from data. This formulation not only leads to greater expressive power but also stronger generalization capability. On two large datasets, Kinetics and NTU-RGBD, it achieves substantial improvements over mainstream methods.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning to Simulate Complex Physics with Graph Networks (2020, background): “…et al., 2019; Sanchez-Gonzalez et al., 2019), as well as non-physical systems, such as multi-agent dynamics (Tacchetti et al., 2018; Sun et al., 2019), algorithm execution (Velikovi et al., 2020), and other dynamic graph settings (Trivedi et al., 2019; 2017; Yan et al., 2018; Manessi et al., 2020).”
Class-Incremental Continual Learning Into the eXtended DER-Verse (2022, neutral): “We read the raw data as done by the ofﬁcial codebase for [76], which represents each sample as a S × T × J × B tensor, where S = 3 is the number of dimensions in space, T = 300 is the number of frames per sequence, J = 25 is the number of tracked skeletal compute the rescaling factor M ← save new…”
Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, neutral): “research domain [26], [27], and a series of hand-crafted features [28], [29], [30], [31], [32], [33], [34], [35], [36], [37] and deep learning-based approaches [18], [19], [38], [39], [40], [41], [42], [43], [44] have been proposed.”
Unbiased Scene Graph Generation From Biased Training (2020, neutral): “— yet on which are our core efforts made [69, 53, 6], then pretend that there is a graph — nothing but a sparse object layout with binary links, and finally shroud it into graph neural networks [63] for merely more contextual object representations [65, 17, 54].”
STGAT: Modeling Spatial-Temporal Interactions for Human Trajectory Prediction (2019, background): “Recently, some meth-ods [27] [40] in the ﬁeld of action recognition have made signiﬁcant progress by applying GNNs to spatial-temporal”
Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction (2020, background): “One of the development related to our work is the ST-GCNN [27].”
Graph Convolutional Networks for Temporal Action Localization (2019, methodology): “Thanks to its effectiveness, GCNs have been successfully applied to several research areas in computer vision, such as skeleton-based action recognition [47], person re-identification [32], and video classification [45].”
Vision GNN: An Image is Worth Graph of Nodes (2022, methodology): “By processing the naturally formed graph of linked human joints, GCN was utilized on human action recognition task [23, 62].”
Gaitgraph: Graph Convolutional Network for Skeleton-Based Gait Recognition (2021, methodology): “Inspired by the success of GCNs in skeleton-based action recognition [9, 10], we adapted the methods to the gait recognition task.”
Amalgamating Knowledge from Heterogeneous Graph Neural Networks (2021, background): “Graph neural networks (GNNs) have achieved unprecedented advances in recent years, showing promising performance in handling graph data lying in the non-Euclidean domain [16, 37, 53, 44, 6, 41, 19, 23, 36, 13, 18, 25, 42].”
Semantic Graph Convolutional Networks for 3D Human Pose Regression (2019, methodology): “We assume that graph data share the same topological structure, such as human skeletons [10, 26, 61, 67], 3D morphable models [33, 45, 72] and citation networks [50].”
Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition (2020, methodology): “, ST-GCN [31], the PA-ResGCN achieves over 10% and 20% relative performance increases with the similar model size on the two datasets, NTU Frame n Frame n+1”
Actional-Structural Graph Convolutional Networks for Skeleton-Based Action Recognition (2019, methodology): “We compare with covering hand-crafted methods [27], RNN/CNNbased deep learning models [6, 22, 20, 14, 21, 13, 18] and recent graph-based methods [29, 26, 23].”
SGCN:Sparse Graph Convolution Network for Pedestrian Trajectory Prediction (2021, background): “To summarize, both Social-STGCNN and SGAN predict overlapping distributions and deviate from the ground-truth, while our predicted distributions exhibit less overlapping and have a better tendency along the ground-truth.”
Skeleton-Based Action Recognition With Shift Graph Convolutional Network (2020, methodology): “For example, ST-GCN [34] costs 16.”
Spectral–Spatial Graph Convolutional Networks for Semisupervised Hyperspectral Image Classification (2019, neutral): “nodes and the corresponding weight values [16].”
MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment (2018, methodology): “Our work is inspired by the GCN [26] and other successful graph-based neural networks [32, 55].”
SR-LSTM: State Refinement for LSTM Towards Pedestrian Trajectory Prediction (2019, methodology): “This work is also inspired by Graph Convolution Networks (GCN) [3, 19] and message passing frameworks used for other applications such as object detection [18, 51], action recognition [34, 46], semantic segmentation [25, 26], scene graph generation [23,24,42,47], video recognition [40], etc .”
Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks (2020, background): “For spatial-temporal graph neural networks, spatial dependencies among nodes are captured by graph convolutions, while temporal dependencies among historical states are preserved by recurrent neural networks [13, 17] or 1D convolutions [22, 23].”