Depth Pooling Based Large-Scale 3-D Action Recognition With Convolutional Neural Networks

By Pichao Wang, Wanqing Li, Zhimin Gao, Chang-Fu Tang, Philip O. Ogunbona. 2018

157 citations recorded by the source corpus.

This paper proposes three simple, compact yet effective representations of depth sequences, referred to respectively as dynamic depth images (DDI), dynamic depth normal images (DDNI), and dynamic depth motion normal images (DDMNI), for both isolated and continuous action recognition. These dynamic images are constructed from a segmented sequence of depth maps using hierarchical bidirectional rank pooling to effectively capture the spatial-temporal information. Specifically, DDI exploits the dynamics of postures over time, and DDNI and DDMNI exploit the 3-D structural information captured by depth maps. Upon the proposed representations, a convolutional neural network (ConvNet)-based method is developed for action recognition. The image-based representations enable us to fine-tune the existing ConvNet models trained on image data without training a large number of parameters from scratch. The proposed method achieved the state-of-art results on three large datasets, namely, the large-scale continuous gesture recognition dataset (means the Jaccard index 0.4109), the large-scale isolated gesture recognition dataset (<inline-formula> <tex-math notation="LaTeX">$\text{59.21}\%$</tex-math></inline-formula>), and the NTU RGB+D dataset (<inline-formula> <tex-math notation="LaTeX">$\text{87.08}\%$</tex-math></inline-formula> cross-subject and <inline-formula> <tex-math notation="LaTeX">$\text{84.22}\%$</tex-math></inline-formula> cross-view) even though only the depth modality was used.

Subfields: Computer Vision and Pattern Recognition, Endocrinology, Diabetes and Metabolism, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Endocrinology, Diabetes and Metabolism

In Human-Computer Interaction

Citations in the corpus

RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “[148] proposed three simple, compact yet effective representations of depth sequences, referred to respectively as Dynamic Depth Images (DDI), Dynamic Depth Normal Images (DDNI) and Dynamic Depth Motion Normal Images (DDMNI), for continuous action recognition.”
MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation (2021, background): “3D human pose estimation (HPE) from monocular videos is a fundamental vision task with a wide range of applications, such as action recognition [22, 23, 39], humancomputer interaction [7], and augmented/virtual reality [30].”
Exploiting Temporal Contexts With Strided Transformer for 3D Human Pose Estimation (2021, background): “dous attention in the past decades [1]–[4] since it plays a significant role in wide applications, such as clinic [5], computer animation [6], action recognition [7]–[16], and human-robot interaction [17], [18].”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “the other methods [9], [56], [217] focused on using depth videos captured by active sensors to recognize actions, due to the availability of low-cost and reliable active sensors like Kinect.”