Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features

By Amin Ullah, Jamil Ahmad, Khan Muhammad, Muhammad Sajjad, Sung Wook Baik. IEEE Access, 2018

672 citations recorded by the source corpus.

Recurrent neural network (RNN) and long short-term memory (LSTM) have achieved great success in processing sequential multimedia data and yielded the state-of-the-art results in speech recognition, digital signal processing, video processing, and text data analysis. In this paper, we propose a novel action recognition method by processing the video data using convolutional neural network (CNN) and deep bidirectional LSTM (DB-LSTM) network. First, deep features are extracted from every sixth frame of the videos, which helps reduce the redundancy and complexity. Next, the sequential information among frame features is learnt using DB-LSTM network, where multiple layers are stacked together in both forward pass and backward pass of DB-LSTM to increase its depth. The proposed method is capable of learning long term sequences and can process lengthy videos by analyzing features for a certain time interval. Experimental results show significant improvements in action recognition using the proposed method on three benchmark data sets including UCF-101, YouTube 11 Actions, and HMDB51 compared with the state-of-the-art action recognition methods.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b5979489e11edd76607c219a8bdc83ba4a88ab38

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

VidTr: Video Transformer Without Convolutions (2021, methodology): “The LSTM [25] was later proposed to model the image feature based on ConvNet features [30, 51, 63].”
Predicting residential energy consumption using CNN-LSTM neural networks (2019, methodology): “designed a model combining CNN and Bi-directional LSTM to recognize human action in video sequences [20].”
Convolutional Neural Networks Based Fire Detection in Surveillance Videos (2018, background): “combating different problems including action recognition [20, 21], pose estimation, image classification [22-26], visual saliency detection, object tracking, image segmentation, scene labeling, object localization, indexing and retrieval [27, 28], and speech processing.”
OFF-ApexNet on Micro-expression Recognition System (2018, methodology): “CNN is one of the deep artiﬁcial neural networks that has been widely used in analyzing visual imagery [49–51].”
A survey of the recent architectures of deep convolutional neural networks (2019, methodology): “Many researchers have used CNN to solve various video processing-related problems (Tong et al. 2015; Frizzi et al. 2016; Shi et al. 2017; Ullah et al. 2017; Wang et al. 2017b).”
Deep Learning for Multigrade Brain Tumor Classification in Smart Healthcare Systems: A Prospective Survey (2020, background): “Indeed, the increased usage of CNNs for several computer vision problems in various domains [33]–[36] motivates adopting them for BTC, particularly for smart health monitoring.”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “Some other works [75], [76], [77] adopted the Bi-directional LSTM, which consists of two independent LSTMs to learn both the forward and backward temporal information, for HAR.”