Differential Recurrent Neural Networks for Action Recognition

By Vivek Veeriah, Naifan Zhuang, Guo-Jun Qi. 2015

492 citations recorded by the source corpus.

The long short-term memory (LSTM) neural network is capable of processing complex sequential information since it utilizes special gating schemes for learning representations from long input sequences. It has the potential to model any time-series or sequential data, where the current hidden state has to be considered in the context of the past hidden states. This property makes LSTM an ideal choice to learn the complex dynamics of various actions. Unfortunately, the conventional LSTMs do not consider the impact of spatio-temporal dynamics corresponding to the given salient motion patterns, when they gate the information that ought to be memorized through time. To address this problem, we propose a differential gating scheme for the LSTM neural network, which emphasizes on the change in information gain caused by the salient motions between the successive frames. This change in information gain is quantified by Derivative of States (DoS), and thus the proposed LSTM model is termed as differential Recurrent Neural Network (dRNN). We demonstrate the effectiveness of the proposed model by automatically recognizing actions from the real-world 2D and 3D human action datasets. Our study is one of the first works towards demonstrating the potential of learning complex time-series representations via high-order derivatives of states.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis (2016, background): “Differential RNN [36] added a new gating mechanism to the traditional LSTM to extract the derivatives of internal state (DoS).”
NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “Recently, deep learning based-approaches have been proposed for 3D human activity analysis [9], [59], [60].”
Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos (2016, background): “They divided the input 3D human skeletal data to five groups of joints and fed them into a separated bidirectional RNN.”
Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, background): “As a result, activity analysis with 3D skeleton data becomes a popular domain of research [9], [10], [11], [12], [13], [14], [15], [16], [17] thanks to its succinctness, high level representation, and robustness against variations in viewpoints, illumination, clothing textures, and”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “[134] proposed a differential gating scheme for the LSTM neural network, which emphasizes the change in information gain caused by the salient motions between the successive frames.”
Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks (2017, background): “[54] proposed a differential gating scheme to make the LSTM network emphasize on the change of information.”
Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates (2017, background): “[38] proposed to add a new gating mechanism for LSTM to model the derivatives of the memory”
Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks (2017, methodology): “[43] present differential RNN that extends LSTM structure by modeling the dynamics of states evolving over time.”
A New Representation of Skeleton Sequences for 3D Action Recognition (2017, methodology): “be used in recurrent neural networks (RNNs) with Long-Short Term Memory (LSTM) neurons [11, 12] to explore the spatial structure and temporal structure of the skeleton sequence for action recognition [6, 41, 54, 37, 26]. Although LSTM networks are designed to explore the long-term temporal dependenc”
Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks (2017, background): “However, using RNNs did not give an improvement over temporal pooling of convolutional features [82] or even over hand-crafted features.”
Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition (2016, background): “Differential LSTM [31] added a new gating inside LSTM to keep track of the derivatives of the memory states in order to discover patterns within salient motion patterns.”
Learning Clip Representations for Skeleton-Based 3D Action Recognition (2018, methodology): “Recently, recurrent neural networks (RNNs) with Long-Short Term Memory (LSTM) neurons [23], [24] have also been used to model the spatial and temporal information of skeleton sequences for action recognition [8]–[11], [25].”
Skeleton Optical Spectra-Based Action Recognition Using Convolutional Neural Networks (2018, background): “Recently, deep learning methods have achieved promising results in various computer vision tasks and how to employ deep learning for action recognition has attracted increasing attention [14]–[17].”
Depth Pooling Based Large-Scale 3-D Action Recognition With Convolutional Neural Networks (2018, background): “The third category is to treat a video as a sequence of images and feed the sequence to a RNN [5]–[7], [14].”
Structured Images for RGB-D Action Recognition (2017, background): “With the recent development of deep learning, a few methods [35, 36, 8, 38, 45, 24] have been developed based on Convolutional Neural Network (ConvNet) or Recurrent Neural Network (RNN).”
Spatio-Temporal Naive-Bayes Nearest-Neighbor (ST-NBNN) for Skeleton-Based Action Recognition (2017, background): “Skeleton-Based Action Recognition In recent years, skeleton-based action recognition problem attracts a lot of attention, and many learning-based methods [5, 24, 35, 21, 14, 13, 15] have been proposed.”