Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep CNN

By Bo Li, Yuchao Dai, Xuelian Cheng, Huahui Chen, Yi Lin, Mingyi He. 2017

259 citations recorded by the source corpus.

We present an image classification based approach to large scale action recognition from 3D skeleton videos. Firstly, we map the 3D skeleton videos to color images, where the transformed action images are translation-scale invariance and dataset independent. Secondly, we propose a multi-scale deep convolutional neural network (CNN) for the image classification task, which could enhance the temporal frequency adjustment of our model. Even though the action images are very different from natural images, the fine-tune strategy still works well. Finally, we exploit various kinds of data augmentation methods to improve the generalization ability of the network. Experimental results on the largest and most challenging benchmark NTU RGB-D dataset show that our method achieves the state-of-the-art performance and outperforms other methods by a large margin.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Skeleton-Based Action Recognition With Directed Graph Neural Networks (2019, methodology): “This design is also similar to the “body parts” conception in previous works for skeleton-based action recognition, which aims to restrict the modeling of joints in a local part of the human body [6, 25, 16].”
Skeleton-based action recognition via spatial and temporal transformer networks (2020, background): “(2) Convolutional neural network (CNN) based methods (Chéron et al. (2015); Simonyan and Zisserman (2014); Ding et al. (2017); Liu et al. (2017c); Li et al. (2017)) leverage spatial information, in a complementary way to RNNbased ones.”
Monocular human pose estimation: A survey of deep learning-based methods (2020, background): “…be applied to many applications such as action / activity recognition (Li et al., 2017b; Luvizon et al., 2018; Li et al., 2018b), action detection (Li et al., 2017a), human tracking (Insafutdinov et al., 2017), Movies and animation, Virtual reality, Human-computer interaction, Video…”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “features learned from skeleton data are not always translation, scale, and rotation invariant, several methods [180], [181] have also been designed to handle these issues.”
Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition (2018, methodology): “Action recognition methods based on skeleton data have been widely investigated and attracted considerable attention due to their strong adaptability to the dynamic circumstance and complicated background [31, 8, 6, 27, 22, 29, 33, 19, 20, 21, 14, 13, 23, 18, 17, 32, 30, 34].”
Spatial Temporal Transformer Network for Skeleton-based Action Recognition (2020, methodology): “(2) CNNbased methods [25]–[29] leverage on spatial information, in a complementary way to RNN-based ones.”
Decoupling GCN with DropGraph Module for Skeleton-Based Action Recognition (2020, methodology): “Deep learning methods usually rearrange a skeleton sequence as a pseudo-image [21,14,9] or a series of joint coordinates [32,50,25], then use CNNs or RNNs to predict action labels.”