Fusing Geometric Features for Skeleton-Based Action Recognition Using Multilayer LSTM Networks

By Songyang Zhang, Yang Yang, Jun Xiao, Xiaoming Liu, Yi Yang, Di Xie, Yueting Zhuang. 2018

163 citations recorded by the source corpus.

Recent skeleton-based action recognition approaches achieve great improvement by using recurrent neural network (RNN) models. Currently, these approaches build an end-to-end network from coordinates of joints to class categories and improve accuracy by extending RNN to spatial domains. First, while such well-designed models and optimization strategies explore relations between different parts directly from joint coordinates, we provide a simple universal spatial modeling method perpendicular to the RNN model enhancement. Specifically, according to the evolution of previous work, we select a set of simple geometric features, and then seperately feed each type of features to a three-layer LSTM framework. Second, we propose a multistream LSTM architecture with a new smoothed score fusion technique to learn classification from different geometric feature streams. Furthermore, we observe that the geometric relational features based on distances between joints and selected lines outperform other features and the fusion results achieve the state-of-the-art performance on four datasets. We also show the sparsity of input gate weights in the first LSTM layer trained by geometric features and demonstrate that utilizing joint-line distances as input require less data for training.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Liu
2. G. Wang
3. Alex ChiChung Kot
4. Ling-yu Duan
5. Amir Shahroudy
6. Hossein Rahmani
7. Qiuhong Ke
8. Zehua Sun

In Biomedical Engineering

1. Jun Liu
2. G. Wang
3. Ling-yu Duan
4. Alex ChiChung Kot
5. Hossein Rahmani
6. Amir Shahroudy
7. Qiuhong Ke
8. Zehua Sun

In Computer Vision and Pattern Recognition

1. Jun Liu
2. G. Wang
3. Hossein Rahmani
4. Alex ChiChung Kot
5. Qiuhong Ke
6. Ling-yu Duan
7. Amir Shahroudy
8. Zehua Sun

In Model Architectures

1. G. Wang
2. Jun Liu
3. Ling-yu Duan
4. Alex ChiChung Kot
5. Amir Shahroudy
6. Qiuhong Ke
7. Hossein Rahmani
8. Mauricio Perez

Citations in the corpus

• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “Specifically, many of them have been evaluated based on the preliminary version [47] of our dataset, or pre-trained on it for transfer learning for other tasks [43], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79].”
• Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “Hence, various methods [155], [156], [157], [158] have applied and adapted RNNs and LSTMs to effectively model the temporal context information within the skeleton sequences for HAR.”
• Multichannel Attention Refinement for Video Question Answering (2020, methodology): “To measure the understanding ability of methods on the video, researchers proposed different intermediate tasks such as video classification [21, 34, 59] and action recognition [28, 35, 37, 64, 65].”