RGB-D-based Human Motion Recognition with Deep Learning: A Survey

By Pichao Wang, Wanqing Li, Philip O. Ogunbona, Jun Wan, Sergio Escalera. 2017

384 citations recorded by the source corpus.

Human motion recognition is one of the most important branches of human-centered research activities. In recent years, motion recognition based on RGB-D data has attracted much attention. Along with the development in artificial intelligence, deep learning techniques have gained remarkable success in computer vision. In particular, convolutional neural networks (CNN) have achieved great success for image-based tasks, and recurrent neural networks (RNN) are renowned for sequence-based problems. Specifically, deep learning methods based on the CNN and RNN architectures have been adopted for motion recognition using RGB-D data. In this paper, a detailed overview of recent advances in RGB-D-based motion recognition is presented. The reviewed methods are broadly categorized into four groups, depending on the modality adopted for recognition: RGB-based, depth-based, skeleton-based and RGB+D-based. As a survey focused on the application of deep learning to RGB-D-based motion recognition, we explicitly discuss the advantages and limitations of existing techniques. Particularly, we highlighted the methods of encoding spatial-temporal-structural information inherent in video sequence, and discuss potential directions for future research.

Subfields: Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “Readers are referred to these survey papers [31], [32], [33], [34], [35], [36] for a more extensive list of the current 3D activity analysis datasets and methods.”
Monocular human pose estimation: A survey of deep learning-based methods (2020, background): “…very important research ﬁeld and can be applied to many applications such as action / activity recognition (Li et al., 2017b; Luvizon et al., 2018; Li et al., 2018b), action detection (Li et al., 2017a), human tracking (Insafutdinov et al., 2017), Movies and animation, Virtual reality,…”
An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition (2019, methodology): “The RGB video based action recognition methods [24, 34, 27, 35] mainly focus on modeling spatial and temporal representations from RGB frames and temporal optical flow.”
Vision-based human activity recognition: a survey (2020, background): “Reviews on deep-learning based methods of human activities recognition were provided in [16, 132, 210, 225].”
Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition (2021, background): “…relatively high-level information for human actions, which exhibits stronger robustness to appearance variation and environmental noises, e.g. , background clutter, illumination changes (Wang et al. 2018; Vemulapalli, Arrate, and Chellappa 2014; Liu, Liu, and Chen 2017; Tu, Liu, and Liu 2018).”
Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition (2019, methodology): “However, most approaches [11, 47] overlook the importance of the semantic information and underexplore it.”