Learning Long-Term Dependencies for Action Recognition with a Biologically-Inspired Deep Network

By Yemin Shi, Yonghong Tian, Yaowei Wang, Wei Zeng, Tiejun Huang. 2016

65 citations recorded by the source corpus.

Despite a lot of research efforts devoted in recent years, how to efficiently learn long-term dependencies from sequences still remains a pretty challenging task. As one of the key models for sequence learning, recurrent neural network (RNN) and its variants such as long short term memory (LSTM) and gated recurrent unit (GRU) are still not powerful enough in practice. One possible reason is that they have only feedforward connections, which is different from the biological neural system that is typically composed of both feedforward and feedback connections. To address this problem, this paper proposes a biologicallyinspired deep network, called shuttleNet. Technologically, the shuttleNet consists of several processors, each of which is a GRU while associated with multiple groups of hidden states. Unlike traditional RNNs, all processors inside shuttleNet are loop connected to mimic the brain’s feedforward and feedback connections, in which they are shared across multiple pathways in the loop connection. Attention mechanism is then employed to select the best information flow pathway. Extensive experiments conducted on two benchmark datasets (i.e UCF101 and HMDB51) show that we can beat state-of-the-art methods by simply embedding shuttleNet into a CNN-RNN framework.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “[112] proposed a biologically-inspired deep network, called ShuttleNet1.”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “tially localize the discriminative information across a video sequence. LSTM has also been used to learn temporal attention to weight features of all frames [80]. Besides using LSTM, some works [82], [83], [84], [85] have used Gated Recurrent Units (GRUs) [86] for HAR, which can also alleviate the v”
Weakly Supervised Action Localization by Sparse Temporal Pooling Network (2017, background): “On the other hand, many algorithms develop techniques to recognize actions based on existing representation methods [40, 42, 8, 11, 9, 26].”
Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition (2017, methodology): “To capture long-term temporal information from videos, one intuitive approach is to introduce the Long Short-Term Memory (LSTM) module as an encoder to encode the relationship between the sequence-illustrating deep features [47, 32, 28].”
Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition (2019, methodology): “Generally speaking, they can be divided into two directions, namely, (1) two-stage solutions [8, 1, 16, 29, 35] which extract spatial feature vectors from video frames and then integrate the obtained local feature sequence into one compact video descriptor for recognition; (2) the 2D [45, 36, 10, 7]”
StNet: Local and Global Spatial-Temporal Modeling for Action Recognition (2018, methodology): “…in videos has received signiﬁcant research attention in the computer vision and machine learning community (Karpathy et al. 2014; Wang and Schmid 2013; Simonyan and Zisserman 2014; Fernando et al. 2015; Wang et al. 2016; Qiu, Yao, and Mei 2017; Carreira and Zisserman 2017; Shi et al. 2017).”
Compressed Video Action Recognition (2017, unknown): “ShuttleNet [30] 94.”