Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Networks

By Pavlo Molchanov, Xiaodong Yang, Shalini Gupta, Kihwan Kim, Stephen Tyree, Jan Kautz. 2016

672 citations recorded by the source corpus.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, 3D Vision & Pose, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

In Human-Computer Interaction

Citations in the corpus

Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “Fusing modalities is traditionally done as late [40], or early fusion [57].”
Pose-conditioned Spatio-Temporal Attention for Human Action Recognition (2017, background): “Sequences of frames are stacked into volumes and fed into convolutional layers at first stages [3, 15, 27, 28, 40].”
A Closer Look at Spatiotemporal Convolutions for Action Recognition (2017, background): “3D CNNs features were also demonstrated to generalize well to other tasks, including action detection [28], video captioning [24], and hand gesture detection [23].”
Pruning Convolutional Neural Networks for Resource Efficient Transfer Learning (2016, background): “Molchanov et al. [10] learn to recognize 25 dynamic hand gestures in streaming video with a large recurrent neural network.”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, background): “Recently, many datasets such as MSCOCO [66] and OpenImage [15] have been proposed for object detection and many ConvNet-based models [1], [2], [3], [67], [68], [69], [70], [71] have been proposed and obtained great performance.”
Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization (2017, methodology): “The neural network methods adopted in gesture analysis include CNNs [17, 18, 26], 3D CNNs [21, 23, 30] and temporal convolutions [26].”
A Unified Framework for Multi-Modal Isolated Gesture Recognition (2018, background): “In other words, existing gesture recognition methods [Molchanov et al. 2016; Molchanov et al. 2015; Pigou et al. 2015] which deal with a limited number of gestures can make very “biased” estimations when it comes to classifying gesture datasets that involve comprehensive inter-and-intra class variat”
Multi-Modality Fusion based on Consensus-Voting and 3D Convolution for Isolated Gesture Recognition (2016, background): “These scores are further fused as the eventual label for isolated gesture recognition.”
A Low Power, Fully Event-Based Gesture Recognition System (2017, neutral): “Convolutional neural networks (CNNs) have been applied successfully to hand gesture recognition and localization [27, 32, 18, 40, 14], as have recurrent neural networks [28].”
First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations (2017, background): “Most previous work in RGB-D action recognition focuses on actions performed by the whole human body with some exceptions that are mainly applicationoriented such as hand gestures for human-computer interaction [7, 14, 24, 32] and sign language [59].”
Pattern Recognition (2018, neutral): “We use temporal data augmentation instead which is common in the action recognition ﬁeld [31,32] .”
Ieee Transactions on Pattern Analysis and Machine Inteligence 1 Super Normal Vector for Human Activity Recognition with Depth Cameras (neutral): “R ECOGNIZING human activities has been widely applied to a number of real-world applications, e.g., humancomputer interaction [26], surveillance event detection [48], content-based video search and summarization [52], etc. Human activity recognition can be performed at various abstract levels.”
Pruning Convolutional Neural Networks for Resource Efficient Inference (2016, background): “Figure 7: Pruning of a recurrent 3D-CNN for dynamic hand gesture recognition (Molchanov et al., 2016).”
Two streams Recurrent Neural Networks for Large-Scale Continuous Gesture Recognition (2016, background): “[19] proposed Recurrent 3D Convolutional Neural Networks (R3DCNN) which performed detection and recognition of gestures from multimodal data simultaneous and supported online recognition with zero or negative lag.”
EgoGesture: A New Dataset and Benchmark for Egocentric Hand Gesture Recognition (2018, methodology): “Another way is to employ sequence labeling models such as RNNs [16] and HMMs [42], [43] to predict the label for the sequence.”
A Deep Neural Framework for Continuous Sign Language Recognition by Iterative Training (2019, methodology): “[4] apply 3D-CNNs for spatiotemporal feature extraction from video streams on color, depth and optical flow”
Large-scale gesture recognition with a fusion of RGB-D data based on the C3D model (2016, methodology): “This method achieves a promising accuracy even on large-scale datasets, and recently many a method are on the basis of it like [20] and [21].”
MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language (2018, methodology): “We can categorize current action recognition or gesture detection into three major categories or combination of them 1) Using 2D convolution on image and do a recurrent network on top of that [20, 25] 2) Extracting subject’s body joints in the form of skeleton and using skeleton data for recognition”