Large-scale Isolated Gesture Recognition using Convolutional Neural Networks

By Pichao Wang, Wanqing Li, Song Liu, Zhimin Gao, Chang-Fu Tang, Philip O. Ogunbona. 2016

98 citations recorded by the source corpus.

This paper proposes three simple, compact yet effective representations of depth sequences, referred to respectively as Dynamic Depth Images (DDI), Dynamic Depth Normal Images (DDNI) and Dynamic Depth Motion Normal Images (DDMNI). These dynamic images are constructed from a sequence of depth maps using bidirectional rank pooling to effectively capture the spatial-temporal information. Such image-based representations enable us to fine-tune the existing ConvNets models trained on image data for classification of depth sequences, without introducing large parameters to learn. Upon the proposed representations, a convolutional Neural networks (ConvNets) based method is developed for gesture recognition and evaluated on the Large-scale Isolated Gesture Recognition at the ChaLearn Looking at People (LAP) challenge 2016. The method achieved 55.57% classification accuracy and ranked 2nd place in this challenge but was very close to the best performance even though we only used depth data.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “For example, optical flow-based methods with Convolutional Neural Networks (CNN) is very effective for RGB channel [27]; depth rank pooling based-method with CNN is a good choice for depth modality [152]; sequence based method with Recurrent Neural Networks (RNN) [82] and image-based method with CNN”
Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks (2017, background): “The last category represents a video in one or multiple compact images and adopt available trained ConvNet architectures for finetuning [51, 52, 56, 1, 9, 53, 54].”
Multi-Modality Fusion based on Consensus-Voting and 3D Convolution for Isolated Gesture Recognition (2016, unknown): “On ChaLearn IsoGD, hand-crafted features such as MFSK [34] as well as its variant which combines DeepID feature [36] scores relatively low compared to deep learning based methods such as AMRL [22] which incorporates”
Review of tool condition monitoring in machining and opportunities for deep learning (2020, methodology): “…owing to the max-pooling layers, the features acquired by the convolution layers The CNN method, which is frequently used in image recognition applications, is also used in gesture recognition [56 – 59], face recognition [60 – 62], scene labeling [63], and action recognition, as well [64, 65].”
Spatially and Temporally Structured Global to Local Aggregation of Dynamic Depth Information for Action Recognition (2018, background): “The last category is to represent video with one or multiple compact images and adopt available trained ConvNet architectures for fine-tuning [14], [15], [17], [29], [42]–[44].”
Multimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM (2017, background): “Available: https://arxiv.org/abs/1502.00873 VOLUME 5, 2017 4523 [20] L. Wang et al. (2016).”
A Survey on Deep Learning Based Approaches for Action and Gesture Recognition in Image Sequences (2017, background): “The authors of [117] use three representations of dynamic depth image (DDI), dynamic depth normal image (DDNI) and dynamic depth motion normal image (DDMNI) for gesture recognition.”
Depth Pooling Based Large-Scale 3-D Action Recognition With Convolutional Neural Networks (2018, methodology): “the bidirectional rank pooling in the method of [20] is replaced with hierarchical and bidirectional rank pooling to capture both high order and non-linear dynamics more effectively; (2) The”
Large-scale gesture recognition with a fusion of RGB-D data based on the C3D model (2016, methodology): “The final score of top 7 results, including [24] and [25] (which wins the second and third place) and the baseline method [2] on testing data are shown in Fig.”
A Unified Framework for Multi-Modal Isolated Gesture Recognition (2018, methodology): “To solve the aforementioned issue, we propose a novel two stream convolutional network (2SCVN) based on the idea of consensus voting adapted from [Simonyan and Zisserman 2014a; Wang et al. 2016; Feichtenhofer et al. 2016b].”