VideoLSTM convolves, attends and flows for action recognition

By Zhenyang Li, Kirill Gavrilyuk, Efstratios Gavves, Mihir Jain, Cees G. M. Snoek. 2016

484 citations recorded by the source corpus.

We present a new architecture for end-to-end sequence learning of actions in video, we call VideoLSTM. Rather than adapting the video to the peculiarities of established recurrent or convolutional architectures, we adapt the architecture to fit the requirements of the video medium. Starting from the soft-Attention LSTM, VideoLSTM makes three novel contributions. First, video has a spatial layout. To exploit the spatial correlation we hardwire convolutions in the soft-Attention LSTM architecture. Second, motion not only informs us about the action content, but also guides better the attention towards the relevant spatio-temporal locations. We introduce motion-based attention. And finally, we demonstrate how the attention from VideoLSTM can be used for action localization by relying on just the action class label. Experiments and comparisons on challenging datasets for action classification and localization support our claims.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

UvA-DARE (Digital Academic Repository) Self-Supervised Video Representation Learning With Odd-One-Out Networks (positive): “Recently, supervised convolutional features obtain state-of-the-art performance either using very large video collections or using 3D convolutions [39] or by fine-tuning ImageNet pre-trained models [5, 16, 30, 43, 46].”
Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “[46, 33]).”
Multiscale Vision Transformers (2021, background): “Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [70, 67, 94, 96, 51, 15, 18, 39, 99, 87, 46] and video [93, 36, 14, 85, 74, 112, 102, 34, 111, 40, 33, 123, 62].”
TSM: Temporal Shift Module for Efficient Video Understanding (2018, background): “Attention mechanism also proves to be effective for temporal modeling [37, 28, 32].”
Spatially Adaptive Computation Time for Residual Networks (2016, methodology): “While we use SACT mechanism for Residual Networks, it can potentially be used for convolutional LSTM [41] models for video processing [28].”
X3D: Expanding Architectures for Efficient Video Recognition (2020, methodology): “These extensions include direct transformation of 2D models [23, 37, 51, 58] such as ResNet or Inception to 3D [6,22,45,61,62,76], adding RNNs on top of 2D CNNs [12, 39, 40, 43, 54, 79], or extending 2D models with an optical flow stream that is processed by an identical 2D network [6, 17, 50, 67] .”
Multimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM (2017, methodology): “VideoLSTM [24] uses convolutional LSTM networks to learn spatiotemporal features from the previously extracted”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, methodology): “Later on, this model is extended to a deeper LSTM for action recognition [74], [75], video captioning [76], and gesture recognition tasks [77].”
Attention mechanisms in computer vision: A survey (2021, unknown): “[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90”
HMS: Hierarchical Modality Selection for Efficient Video Recognition (2021, background): “The former line of research aggregates spatial features along the temporal axis with pooling operations [6, 33, 41], recurrent networks like long short-term memory (LSTM) network [17, 24] or carefully designed modules like TRN [53] and TSM [18].”
Lattice Long Short-Term Memory for Human Action Recognition (2017, methodology): “To address this, in VideoLSTM [19] convolutions are hardwired in the soft-Attention LSTM [26].”
Flow-Guided Feature Aggregation for Video Object Detection (2017, methodology): “On one hand, most of these work [34, 24, 47, 7, 46, 1, 9, 35] use recurrent neural network (RNNs) to aggregate features from consecutive frames.”
Predicting Human Eye Fixations via an LSTM-Based Saliency Attentive Model (2016, background): “In this last non task-driven context, neural attention has been successfully applied to image captioning [16] and machine translation [17] to selectively focus on different parts of a sentence, and to action recognition [18] to focus on the relevant parts of a spatio-temporal volume.”