View Adaptive Neural Networks for High Performance Skeleton-Based Human Action Recognition

By Pengfei Zhang, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jianru Xue, Nanning Zheng. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018

490 citations recorded by the source corpus.

Skeleton-based human action recognition has recently attracted increasing attention thanks to the accessibility and the popularity of 3D skeleton data. One of the key challenges in action recognition lies in the large variations of action representations when they are captured from different viewpoints. In order to alleviate the effects of view variations, this paper introduces a novel view adaptation scheme, which automatically determines the virtual observation viewpoints over the course of an action in a learning based data driven manner. Instead of re-positioning the skeletons using a fixed human-defined prior criterion, we design two view adaptive neural networks, i.e., VA-RNN and VA-CNN, which are respectively built based on the recurrent neural network (RNN) with the Long Short-term Memory (LSTM) and the convolutional neural network (CNN). For each network, a novel view adaptation module learns and determines the most suitable observation viewpoints, and transforms the skeletons to those viewpoints for the end-to-end recognition with a main classification network. Ablation studies find that the proposed view adaptive models are capable of transforming the skeletons of various views to much more consistent virtual viewpoints. Therefore, the models largely eliminate the influence of the viewpoints, enabling the networks to focus on the learning of action-specific features and thus resulting in superior performance. In addition, we design a two-stream scheme (referred to as VA-fusion) that fuses the scores of the two networks to provide the final prediction, obtaining enhanced performance. Moreover, random rotation of skeleton sequences is employed to improve the robustness of view adaptation models and alleviate overfitting during training. Extensive experimental evaluations on five challenging benchmarks demonstrate the effectiveness of the proposed view-adaptive networks and superior performance over state-of-the-art approaches.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/a58027f532153b9f9c6ad884d85a175862fc16e6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Jun Liu
2. Liang Wang
3. Qiuhong Ke
4. Hossein Rahmani
5. Nanning Zheng
6. Pengfei Zhang
7. Jianru Xue
8. Zhang Zhang

In Computer Vision and Pattern Recognition

1. Jun Liu
2. Qiuhong Ke
3. Hossein Rahmani
4. Liang Wang
5. G. Wang
6. Wenjun Zeng
7. Cuiling Lan
8. Zehua Sun

In Human-Computer Interaction

1. Liang Wang
2. Caifeng Shan
3. Yisheng Song
4. Zhang Zhang
5. Jun Liu
6. Qiuhong Ke
7. Nanning Zheng
8. Wenjun Zeng

Citations in the corpus

• Constructing Stronger and Faster Baselines for Skeleton-Based Action Recognition (2021, background): “The study in [30] designs a view adaptive model to automatically regulate observation viewpoints during the occurrence of an action, so as to obtain view invariant representations of human actions.”
• Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition (2019, methodology): “In recent years, skeleton-based action recognition [56, 7, 36, 58] is attracting increasing interests.”
• Human Action Recognition From Various Data Modalities: A Review (2020, background): “For example, the works of [171], [178], [179] focused on handling the viewpoint variation issue.”
• Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition (2020, methodology): “The study in [34] designs a view adaptive model to automatically regulate observation viewpoints during the occurrence of an action, so as to obtain the view invariant representations of human actions.”