Action Recognition using Visual Attention

By Shikhar Sharma, Ryan Kiros, Ruslan Salakhutdinov. 2015

684 citations recorded by the source corpus.

We propose a soft attention based model for the task of action recognition in videos. We use multi-layered Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units which are deep both spatially and temporally. Our model learns to focus selectively on parts of the video frames and classifies videos after taking a few glimpses. The model essentially learns which parts in the frames are relevant for the task at hand and attaches higher importance to them. We evaluate the model on UCF-11 (YouTube Action), HMDB-51 and Hollywood2 datasets and analyze how the model focuses its attention depending on the scene and the action being performed.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Spatiotemporal Residual Networks for Video Action Recognition (2016, methodology): “Yet other work uses a multilayer LSTM to let the model attend to relevant spatial parts in the input frames [19].”
Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “Soft attention was proposed for image [9, 58] and video understanding [46, 48, 59] with spatial, temporal, and spatio-temporal variants.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “[111] proposed a soft attention model for action recognition based on LSTM (see Figure.”
Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks (2017, methodology): “Moreover, a recurrent attention mechanism is proposed to iteratively promote the attention capability of our network, while the methods in [60], [68], and [69] performed attention only once.”
Pose-conditioned Spatio-Temporal Attention for Human Action Recognition (2017, background): “In this regard, models of visual attention [26, 7, 33] (see section 2 for a full discussion) have drawn considerable interest recently.”
Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks (2017, background): “However, using RNNs did not give an improvement over temporal pooling of convolutional features [82] or even over hand-crafted features.”
TSM: Temporal Shift Module for Efficient Video Understanding (2018, background): “Attention mechanism also proves to be effective for temporal modeling [37, 28, 32].”
Video Action Transformer Network (2018, background): “Attention for videos has been pursued in various forms, including gating or second order pooling [10, 27, 28, 45] guided by human pose or other primitives [4, 5, 10, 11], recurrent models [34] and self-attention [44].”
Spatially Adaptive Computation Time for Residual Networks (2016, background): “On the other hand, soft spatial attention models [42, 36] do not allow to save computation since they require evaluating the model at all spatial positions to choose per-position attention weights.”
VideoLSTM convolves, attends and flows for action recognition (2016, neutral): “Before discussing the impact of flow, we first note that both the original ALSTM (Sharma et al., 2016) and ConvLSTM models (Shi et al.”
A-Fast-RCNN: Hard Positive Generation via Adversary for Object Detection (2017, methodology): “Also note that our output masks are different from the Attention Mask proposed in [31], where they use the attention mechanism to facilitate classification.”
Emergent linguistic structure in artificial neural networks trained by self-supervision (2020, background): “Attention has been a highly successful neural network component for processing text (31), video (32), image (33), and speech (34) data.”
Understanding Patch-Based Learning by Explaining Predictions (2018, background): “Since such label noise makes the training more difﬁcult [18], recent work investigated ways to cope with this problem, e.g., attention mechanisms [22] or weighted patch aggregation [2].”
Differentiable Patch Selection for Image Recognition (2021, background): “While this helps the model focus on relevant features [37, 50, 42, 26], become more interpretable [26], or include external data such as image captions [46, 2], the models will typically still process the whole image on a fixed input resolution.”
Video-based emotion recognition using CNN-RNN and C3D hybrid networks (2016, methodology): “Deep-face VGG model (VGG-FACE) [14] is tested for comparisons with VGG model trained on ImageNet.”
Body Joint Guided 3-D Deep Convolutional Descriptors for Action Recognition (2017, background): “It should be noted that the recently popular concept of soft attention-based model [34], [35] can also fit into the”