An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data

By Sijie Song, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jiaying Liu. 2016

1,062 citations recorded by the source corpus.

Human action recognition is an important task in computer vision. Extracting discriminative spatial and temporal features to model the spatial and temporal evolutions of different actions plays a key role in accomplishing this task. In this work, we propose an end-to-end spatial and temporal attention model for human action recognition from skeleton data. We build our model on top of the Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM), which learns to selectively focus on discriminative joints of skeleton within each frame of the inputs and pays different levels of attention to the outputs of different frames. Furthermore, to ensure effective training of the network, we propose a regularized cross-entropy loss to drive the model learning process and develop a joint training strategy accordingly. Experimental results demonstrate the effectiveness of the proposed model, both on the small human action recognition dataset of SBU and the currently largest NTU dataset.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Jun Liu
2. Liang Wang
3. Hanqing Lu
4. Yifan Zhang
5. Jian Cheng
6. G. Wang
7. Hossein Rahmani
8. Wanqing Li

In Computer Vision and Pattern Recognition

1. Jun Liu
2. Luc Van Gool
3. G. Wang
4. Dacheng Tao
5. Ming-Hsuan Yang
6. Liang Wang
7. Xiaogang Wang
8. Yi Yang

In Model Architectures

1. Hanqing Lu
2. Yifan Zhang
3. Jian Cheng
4. G. Wang
5. Jun Liu
6. Alex ChiChung Kot
7. Ling-yu Duan
8. Amir Shahroudy

Citations in the corpus

• Video Representation Learning Using Discriminative Pooling (2018, neutral): “…26.7% 14.2% NTU-RGBD Method Cross-Subject Cross-View Res-TCN (Average Pooling)[20] 74.3% 83.1% Res-TCN (Rank Pooling [2]) 75.5% 83.9% STA-LSTM [44] 73.4% 81.2% ST-LSTM + Trust Gate[28] 69.2% 77.7% Body-parts learning [33] 75.2% 83.1% SVMP (Res-TCN) 78.5% 86.4% of these methods could enjoy a…”
• 2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning (2018, methodology): “To cope with noisy skeletons, Spatio-Temporal LSTM networks have been widely used by applying a gating mechanism [26] to learn the reliability of skeleton sequences or by using attention mechanisms [27, 46].”
• Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “Soft attention was proposed for image [9, 58] and video understanding [46, 48, 59] with spatial, temporal, and spatio-temporal variants.”
• RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “Most of available datasets require subjects to perform actions in a visible and restricted view to avoid occlusion, and this results in limited view data collection and less 29 SBU Kinect Interaction [149] Depth+ Skeleton PRNN Side Information 89.20% [130] Skeleton Co-occurrence +LSTM None 90.41% [1”
• Interpretable 3D Human Action Analysis with Temporal Convolutional Networks (2017, methodology): “In our proposed version of TCN, we show that our model also learns both spatial and temporal attention without the need for initial pre-training stage as in [31].”
• Pose-conditioned Spatio-Temporal Attention for Human Action Recognition (2017, background): “An attention mechanism on poses could be an option, similar to [34].”
• View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data (2017, methodology): “In comparison with F-trans & F-rota + LSTM [31, 24, 35], our scheme achieves improvement by 5.3% and 3.7% in accuracy for CS and CV settings, respectively.”
• PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding (2017, background): “Sliding Window + STA-LSTM: Spatial-temporal attention network [33] is a state-of-the-art work proposed for action recognition with unidirectional LSTM.”
• An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition (2019, methodology): “A spatial-temporal attention network learns to selectively focus on discriminative spatial and temporal features in [25].”
• Cross-Modal Knowledge Distillation for Action Recognition (2019, background): “More recently, most approaches use either recurrent neural networks to learn spatio-temporal features from sequences of skeleton data [14, 15, 12] or convolutional neural networks for classifying the skeleton sequences [16, 8, 17].”
• WALKING WALKing walking: Action Recognition from Action Echoes (2017, methodology): “Popular dynamics-based methods include: Hidden Markov Models (HMMs) [X-ia et al. , 2012; Wu and Shao, 2014], Long-Short Term Memory networks (LSTM) [Du et al. , 2015; Zhu et al. , 2016; Song et al. , 2016].”
• SSNet: Scale Selection Network for Online 3D Action Prediction (2018, background): “As a result, activity analysis with 3D skeleton data becomes popular [19, 49, 62, 41, 81, 57, 43] due to its succinctness, high level representation, and robustness against variations in viewpoints, clothing textures, illumination, and background [11, 42, 23].”
• Spatio-Temporal Graph Convolution for Skeleton Based Action Recognition (2018, methodology): “When increasing K 1 , K 2 , the local filtering region will cover (Vemulapalli, Arrate, and Chellappa 2014) 52.76% 50.08% Dynamic Skeletons (Hu et al. 2015) 65.22% 60.23% HBRNN (Du, Wang, and Wang 2015) 63.97% 59.07% LieNet (Huang et al. 2016) 66.95% 61.37% Deep LSTM (Shahroudy et al. 2016) 67.29% 60”
• Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning (2018, methodology): “To model the discriminative features, a spatial-temporal attention model [26] based on LSTM is proposed to focus on discriminative joints and pay different attentions to different frames.”