Lattice Long Short-Term Memory for Human Action Recognition

By Lin Sun, K. Jia, Kevin Chen, Dit-Yan Yeung, Bertram E. Shi, Silvio Savarese. 2017

174 citations recorded by the source corpus.

Human actions captured in video sequences are threedimensional signals characterizing visual appearance and motion dynamics. To learn action patterns, existing methods adopt Convolutional and/or Recurrent Neural Networks (CNNs and RNNs). CNN based methods are effective in learning spatial appearances, but are limited in modeling long-term motion dynamics. RNNs, especially Long Short- Term Memory (LSTM), are able to learn temporal motion dynamics. However, naively applying RNNs to video sequences in a convolutional manner implicitly assumes that motions in videos are stationary across different spatial locations. This assumption is valid for short-term motions but invalid when the duration of the motion is long.,,In this work, we propose Lattice-LSTM (L2STM), which extends LSTM by learning independent hidden state transitions of memory cells for individual spatial locations. This method effectively enhances the ability to model dynamics across time and addresses the non-stationary issue of long-term motion dynamics without significantly increasing the model complexity. Additionally, we introduce a novel multi-modal training procedure for training our network. Unlike traditional two-stream architectures which use RGB and optical flow information as input, our two-stream model leverages both modalities to jointly train both input gates and both forget gates in the network rather than treating the two streams as separate entities with no information about the other. We apply this end-to-end system to benchmark datasets (UCF-101 and HMDB-51) of human action recognition. Experiments show that on both datasets, our proposed method outperforms all existing ones that are based on LSTM and/or CNNs of similar model complexities.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “To relieve the drawbacks of 3D kernels, Sun et al. (2015) factorized a 3D filter into a combination of 2D and 1D filters.”
Long-Term Feature Banks for Detailed Video Understanding (2018, methodology): “Temporal and relationship models include RNNs that model the evolution of video frames [7, 24, 27, 44, 57] and multilayer perceptrons that model ordered frame features [58].”
MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, background): “Some works explore the use of 2D networks for video recognition by processing videos in smaller segments followed by late fusion [31, 15, 74, 68, 20, 58, 36, 39, 69, 75, 76].”
Chinese NER Using Lattice LSTM (2018, methodology): “They have been used to model motion dynamics (Sun et al., 2017), dependencydiscourse DAGs (Peng et al.”
X3D: Expanding Architectures for Efficient Video Recognition (2020, methodology): “These extensions include direct transformation of 2D models [23, 37, 51, 58] such as ResNet or Inception to 3D [6,22,45,61,62,76], adding RNNs on top of 2D CNNs [12, 39, 40, 43, 54, 79], or extending 2D models with an optical flow stream that is processed by an identical 2D network [6, 17, 50, 67] .”
Object Level Visual Reasoning in Videos (2018, background): “In activity recognition, they have gained some traction in recent years, either as soft-attention on articulated pose (joints) [36], on feature map cells [34,39], on time [45] or on parts in raw RGB input through diﬀerentiable crops [3].”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “Thus, most of the existing methods have adopted gated RNN architectures, such as Long-Short Term Memory (LSTM) [67], [68], [69], [70], to”
PoTion: Pose MoTion Representation for Action Recognition (2018, unknown): “Overall, we outperform all existing approaches on all datasets, including methods that leverage pose [13, 48] or capture long-term motion [36].”
Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition (2017, methodology): “To capture long-term temporal information from videos, one intuitive approach is to introduce the Long Short-Term Memory (LSTM) module as an encoder to encode the relationship between the sequence-illustrating deep features [47, 32, 28].”
Compressed Video Action Recognition (2017, methodology): “LRCN [6], Composite LSTM Model [35], and L(2)STM [37] use RNNs to model temporal dynamics.”