Spatial Temporal Transformer Network for Skeleton-based Action Recognition
By Chiara Plizzari, Marco Cannici, Matteo Matteucci. 2020
261 citations recorded by the source corpus.
Skeleton-based Human Activity Recognition has achieved a great interest in recent years, as skeleton data has been demonstrated to be robust to illumination changes, body scales, dynamic camera views and complex background. In particular, Spatial-Temporal Graph Convolutional Networks (ST-GCN) demonstrated to be effective in learning both spatial and temporal dependencies on non-Euclidean data such as skeleton graphs. Nevertheless, an effective encoding of the latent information underlying the 3D skeleton is still an open problem, especially how to extract effective information from joint motion patterns and their correlations. In this work, we propose a novel Spatial-Temporal Transformer network (ST-TR) which models dependencies between joints using the Transformer self-attention operator. In our ST-TR model a Spatial Self-Attention module (SSA) is used to understand intra-frame interactions between different body parts, and a Temporal Self-Attention module (TSA) to model inter-frame correlations. The two are combined in a two-stream network, whose performance is evaluated on three large-scale datasets, NTU-RGB+D 60, NTU-RGB+D 120 and Kinetics Skeleton 400, outperforming the state-of-the-art on NTU-RGB+D w.r.t. models using the same input data consisting of joint information.
Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Hong Liu
- Angela Yao
- Juncheng Li
- Dipika Singhania
- Dibyadip Chatterjee
- Linlin Zhang
- Robert Wang
- Kun He
In Biomedical Engineering
In Computer Vision and Pattern Recognition
- Fadime Sener
- Angela Yao
- Dipika Singhania
- Hong Liu
- Juncheng Li
- Dibyadip Chatterjee
- Linlin Zhang
- Robert Wang
Citations in the corpus
- Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities (2022, methodology): “We apply two competing state-of-the-art temporal convolutional networks: MS-TCN++ [24] and C2F-TCN [44], using frame-wise features extracted from TSM [25] trained for action recognition on Assembly as input.”
- Efficient Transformer for Single Image Super-Resolution (2021, methodology): “By adapting Transformer in vision tasks, it has been successfully applied in image recognition [24], [25], object detection [26], [27], low-level image processing [28], [29], action recognition [30], [31] and so on.”