Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video

By Lionel Pigou, Aäron van den Oord, Sander Dieleman, Mieke Van Herreweghe, Joni Dambre. 2015

282 citations recorded by the source corpus.

Recent studies have demonstrated the power of recurrent neural networks for machine translation, image captioning and speech recognition. For the task of capturing temporal structure in video, however, there still remain numerous open research questions. Current research suggests using a simple temporal feature pooling strategy to take into account the temporal aspect of video. We demonstrate that this method is not sufficient for gesture recognition, where temporal information is more discriminative compared to general video classification tasks. We explore deep architectures for gesture recognition in video and propose a new end-to-end trainable neural network architecture incorporating temporal convolutions and bidirectional recurrence. Our main contributions are twofold; first, we show that recurrence is crucial for this task; second, we show that adding temporal convolutions leads to significant improvements. We evaluate the different approaches on the Montalbano gesture recognition dataset, where we achieve state-of-the-art results.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Deep Learning for IoT Big Data and Streaming Analytics: A Survey (2017, methodology): “In [103], a combined model of CNN and RNN was proposed for gesture recognition in video frames.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “[98] directly considered the depth as the fourth channel and CNN was adopted to extract frame-based appearance features.”
Large-scale Isolated Gesture Recognition using pyramidal 3D convolutional networks (2016, background): “The effective spatial convolutional features of hands may be overwhelmed by complex scene backgrounds due to the relative small size of fingers in images; thus the temporal information becomes more discriminative for gesture recognition, compared to the general video classification tasks [17].”
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition (2016, methodology): “, millions of frames in computer vision [22]), they may actually be applicable to problem domains where acquiring annotated data is very costly, such as in supervised activity recognition.”
A Survey on Deep Learning Based Approaches for Action and Gesture Recognition in Image Sequences (2017, methodology): “[42] uses an LSTM to model each individual’s actions and a second-level LSTM aggregates the outputs of individual LSTMs.”
A Deep Neural Framework for Continuous Sign Language Recognition by Iterative Training (2019, methodology): “[21] propose an end-to-end neural model with temporal convolutions and bidirectional recurrence for sign spotting, which is taken as framewise classification in their framework.”
Using Convolutional 3D Neural Networks for User-independent continuous gesture recognition (2016, methodology): “Inspired by the recent progress in the field of deep learning 2D Convolutional Neural Networks (2D CNNs) have been applied to the Gesture Recognition field in order to extract spatial features [7], [8].”
SubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition (2017, background): “The emergence of modern deep learning methods [30, 13, 34] has removed the need for such tailored representations and enabled systems to implicitly learn both the spatial and the temporal features.”
Recurrent Convolutional Network for Video-Based Person Re-identification (2016, background): “These networks include feedback connections that allow the recall of events over time [33], and temporal-pooling networks, that average spatial features over multiple time-steps [34].”
Pattern Recognition (2018, unknown): “We use temporal data augmentation instead which is common in the action recognition ﬁeld [31,32] .”
Multi-Modality Fusion based on Consensus-Voting and 3D Convolution for Isolated Gesture Recognition (2016, background): “Many methods have been proposed over the last few years [23, 14, 13, 36], which can be classified mainly into two categories: continuous gesture recognition and isolated gesture recognition.”
Sensor-based and vision-based human activity recognition: A comprehensive survey (2020, background): “In another research, Pigou et al. examined deep learning for HAR in video and introduced a novel deep neural network architecture that incorporated bidirectional recurrence and temporal convolu-tions [13] .”
Multimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM (2017, background): “[34] explored five kinds of deep architectures for gesture recognition in video, and showed that LRCN-style networks are not optimal for gesture recognition.”