Multimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM

By Guangming Zhu, L. Zhang, Peiyi Shen, Juan Song. 2017

251 citations recorded by the source corpus.

Gesture recognition aims to recognize meaningful movements of human bodies, and is of utmost importance in intelligent human–computer/robot interactions. In this paper, we present a multimodal gesture recognition method based on 3-D convolution and convolutional long-short-term-memory (LSTM) networks. The proposed method first learns short-term spatiotemporal features of gestures through the 3-D convolutional neural network, and then learns long-term spatiotemporal features by convolutional LSTM networks based on the extracted short-term spatiotemporal features. In addition, fine-tuning among multimodal data is evaluated, and we find that it can be considered as an optional skill to prevent overfitting when no pre-trained models exist. The proposed method is verified on the ChaLearn LAP large-scale isolated gesture data set (IsoGD) and the Sheffield Kinect gesture (SKIG) data set. The results show that our proposed method can obtain the state-of-the-art recognition accuracy (51.02% on the validation set of IsoGD and 98.89% on SKIG).

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “[178] RGB+ Depth C3D+ConvLSTM Score Fusion 51.”
A deep-learning-based surrogate model for data assimilation in dynamic subsurface flow problems (2019, unknown): “The convLSTM net and its variants have been used in a range of application areas, including precipitation forecasting [24], video gesture recognition [32] and MRI cardiac segmentation [33], where they have been shown to be effective in capturing both spatial and temporal information.”
MMTM: Multimodal Transfer Module for CNN Fusion (2019, background): “Furthermore, some CNN-based models utilize recurrent architectures to capture the temporal information [72, 9, 12, 76].”
Task-Agnostic Dynamics Priors for Deep Reinforcement Learning (2019, methodology): “Our frame prediction model is closest in spirit to the ConvolutionalLSTM model which has been applied to several domains (Xingjian et al., 2015; Zhu et al., 2017; Ke et al., 2017).”