Multi-Modality Fusion based on Consensus-Voting and 3D Convolution for Isolated Gesture Recognition
By Jiali Duan, Shuai Zhou, Jun Wan, Xiaoyuan Guo, S. Li. 2016
39 citations recorded by the source corpus.
Recently, the popularity of depth-sensors such as Kinect has made depth videos easily available while its advantages have not been fully exploited. This paper investigates, for gesture recognition, to explore the spatial and temporal information complementarily embedded in RGB and depth sequences. We propose a convolutional twostream consensus voting network (2SCVN) which explicitly models both the short-term and long-term structure of the RGB sequences. To alleviate distractions from background, a 3d depth-saliency ConvNet stream (3DDSN) is aggregated in parallel to identify subtle motion characteristics. These two components in an unified framework significantly improve the recognition accuracy. On the challenging Chalearn IsoGD benchmark, our proposed method outperforms the first place on the leader-board by a large margin (10.29%) while also achieving the best result on RGBD-HuDaAct dataset (96.74%). Both quantitative experiments and qualitative analysis shows the effectiveness of our proposed framework and codes will be released to facilitate future research.
Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Biomedical Engineering
In Computer Vision and Pattern Recognition
- Pichao Wang
- Wanqing Li
- Philip O. Ogunbona
- Sergio Escalera
- Jun Wan
- Zhimin Gao
- Peiyi Shen
- Guangming Zhu
In Human-Computer Interaction
- Pichao Wang
- Philip O. Ogunbona
- Wanqing Li
- Sergio Escalera
- Jun Wan
- Peiyi Shen
- Guangming Zhu
- Juan Song
Citations in the corpus
- RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “[27] RGB+ Depth 2SCVN-3DDSN Score Fusion 49.”
- Multimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM (2017, background): “Convolutional neural networks (CNN) and RNN based methods have demonstrated the state-of-the-art performances on human gesture/action recognition [6], [32], [33].”