View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data

By Pengfei Zhang, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jianru Xue, Nanning Zheng. 2017

619 citations recorded by the source corpus.

Skeleton-based human action recognition has recently attracted increasing attention due to the popularity of 3D skeleton data. One main challenge lies in the large view variations in captured human actions. We propose a novel view adaptation scheme to automatically regulate observation viewpoints during the occurrence of an action. Rather than re-positioning the skeletons based on a human defined prior criterion, we design a view adaptive recurrent neural network (RNN) with LSTM architecture, which enables the network itself to adapt to the most suitable observation viewpoints from end to end. Extensive experiment analyses show that the proposed view adaptive RNN model strives to (1) transform the skeletons of various views to much more consistent viewpoints and (2) maintain the continuity of the action rather than transforming every frame to the same position with the same body orientation. Our model achieves significant improvement over the state-of-the-art approaches on three benchmark datasets.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Model Architectures, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Liang Wang
2. Hanqing Lu
3. Yifan Zhang
4. Jian Cheng
5. Jun Liu
6. G. Wang
7. Alex ChiChung Kot
8. Ling-yu Duan

In Computer Vision and Pattern Recognition

1. Jun Liu
2. G. Wang
3. Liang Wang
4. Hanqing Lu
5. Jian Cheng
6. Alex ChiChung Kot
7. Yifan Zhang
8. Amir Shahroudy

In Model Architectures

1. G. Wang
2. Hanqing Lu
3. Yifan Zhang
4. Jian Cheng
5. Jun Liu
6. Alex ChiChung Kot
7. Ling-yu Duan
8. Amir Shahroudy

In Human-Computer Interaction

1. Yifan Zhang
2. Hanqing Lu
3. Jian Cheng
4. Liang Wang
5. Jun Liu
6. Lei Shi
7. Tien-Ping Tan
8. G. Wang

Citations in the corpus

• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “Zhang et al. [80] added a view-adaptation scheme to the LSTM network to regulate the observation viewpoints.”
• Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, background): “is informative enough for representing human behavior, even without appearance information [6].”
• Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “2 VA-LSTM [63] - 79.”
• Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition (2021, methodology): “HBRNN (Du et al. 2015) 64.0 59.1 P-LSTM (Shahroudy et al. 2016) 67.3 60.7 TCN (Soo Kim and Reiter 2017) 83.1 74.3 VA-LSTM (Zhang et al. 2017) 87.7 79.2 ST-GCN (Yan, Xiong, and Lin 2018) 88.3 81.5 AS-GCN (Li et al. 2019a) 94.2 86.8 2s AGC-LSTM (Si et al. 2019) 95.0 89.2 2s AGCN (Shi et al. 2019c) 95.”
• Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveys (2023, background): “gence is the mentor of machine intelligence, and the latter will learn problem-solving strategies from human behaviors, so as to improve the reliability of intelligent systems [163], [164].”
• Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation (2018, methodology): “Since the recognition task on SBU is much easier than on NTU RGB+D, the best accuracy reported has achieved 97.6% [Zhang et al. , 2017].”
• Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition (2018, methodology): “Action recognition methods based on skeleton data have been widely investigated and attracted considerable attention due to their strong adaptability to the dynamic circumstance and complicated background [31, 8, 6, 27, 22, 29, 33, 19, 20, 21, 14, 13, 23, 18, 17, 32, 30, 34].”
• Skeleton-based action recognition via spatial and temporal transformer networks (2020, neutral): “2 VA-LSTM [54] 79.”
• View Adaptive Neural Networks for High Performance Skeleton-Based Human Action Recognition (2018, background): “It should be mentioned that this paper is an extension of our previous conference paper [58].”
• Deep Progressive Reinforcement Learning for Skeleton-Based Action Recognition (2018, background): “Further more, we discover that our proposed method outperforms all the other state-of-the-art methods except VA-LSTM [23].”
• Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition (2020, background): “Considering the sequence property, Recurrent Neural Networks (RNNs) are a natural choice and have been widely studied [18, 23, 34, 35, 37].”
• Skeleton-Based Action Recognition With Shift Graph Convolutional Network (2020, methodology): “7 VA-LSTM [35] 87.”
• Human Action Recognition From Various Data Modalities: A Review (2020, neutral): “Other LSTM-based methods include [201], [206], [246].”
• An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition (2019, methodology): “…Two-stream RNN [30] 2017 79.5 71.3 STA-LSTM [25] 2017 81.2 73.4 Ensemble TS-LSTM [13] 2017 81.3 74.6 Visualization CNN [17] 2017 82.6 76.0 VA-LSTM [41] 2017 87.6 79.4 ST-GCN [39] 2018 88.3 81.5 SR-TSL [22] 2018 92.4 84.8 HCN [14] 2018 91.1 86.5 PB-GCN [26] 2018 93. of two protocols on the NTU…”
• Skeleton-Based Action Recognition With Multi-Stream Adaptive Graph Convolutional Networks (2019, methodology): “Early deep-learning-based approaches for skeleton-based action recognition manually structure the skeleton data as a sequence of joint-coordinate vectors [8], [9], [10], [11], [12], [13], [14] or as a pseudo-image [15], [16], [17], [18], [19], which is fed into RNNs or CNNs to”
• Skeleton-based action recognition with convolutional neural networks (2017, background): “Recently, [7] developed a view adaptive RNN to cope with the viewpoint variations explicitly.”
• Decoupling GCN with DropGraph Module for Skeleton-Based Action Recognition (2020, background): “X-sub X-view Lie Group [39] 50.1 52.8 STA-LSTM [36] 73.4 81.2 VA-LSTM [48] 79.2 87.7 ARRN-LSTM [16] 80.7 88.8 Ind-RNN [20] 81.8 88.0 2-Stream 3DCNN [21] 66.8 72.6 TCN [11] 74.3 83.1 ClipCNN+MTLN [9] 79.6 84.8 Synthesized CNN [27] 80.0 87.2 CNN+Motion+Trans [15] 83.2 88.8 ST-GCN [46] 81.5 88.3…”
• Skeleton-Based Action Recognition With Directed Graph Neural Networks (2019, methodology): “With the development of deep-learning-based methods, the data-driven methods have become the mainstream [7, 25, 20, 28, 37, 18, 19, 14, 13, 21, 17, 16, 34, 29, 3].”