Human Action Recognition: Pose-Based Attention Draws Focus to Hands

By Fabien Baradel, Christian Wolf, Julien Mille. 2017

122 citations recorded by the source corpus.

We propose a new spatio-temporal attention based mechanism for human action recognition able to automatically attend to most important human hands and detect the most discriminative moments in an action. Attention is handled in a recurrent manner employing Recurrent Neural Network (RNN) and is fully-differentiable. In contrast to standard soft-attention based mechanisms, our approach does not use the hidden RNN state as input to the attention model. Instead, attention distributions are drawn using external information: human articulated pose. We performed an extensive ablation study to show the strengths of this approach and we particularly studied the conditioning aspect of the attention mechanism. We evaluate the method on the largest currently available human action recognition dataset, NTU-RGB+D, and report state-of-the-art results. Another advantage of our model are certains aspects of explanability, as the spatial and temporal attention distributions at test time allow to study and verify on which parts of the input data the method focuses.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “9 - STA-Hands [5] X X 82.”
Video Action Transformer Network (2018, background): “Attention for videos has been pursued in various forms, including gating or second order pooling [10, 27, 28, 45] guided by human pose or other primitives [4, 5, 10, 11], recurrent models [34] and self-attention [44].”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “gated in [314], [316], where the classification scores from a CNN model [316] or an RNN model [314] trained on skeleton data and a skeleton conditioned spatio-temporal attention network trained on RGB data were fused for final”
Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition (2020, background): “obviously lower than other SOTA models. Attention Models. Attention mechanisms have become an integral part of compelling sequence modeling in various tasks, such as action recognition. Baradel et al.[2] introduce the attention mechanism into an RGB-based action recognition model, which uses human p”
An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition (2019, background): “Recent works mainly learn human action representations with deep learning networks[40, 37, 2].”
Constructing Stronger and Faster Baselines for Skeleton-Based Action Recognition (2021, methodology): “[39] introduce the attention mechanism into an RGB-based action recognition model, which uses human poses to calculate spatial”
Egocentric Hand Track and Object-Based Human Action Recognition (2019, methodology): “Our idea relates to methods that reduce RGB images to trajectories or poses of hands or objects in the scene and use this contextual information alone for human action recognition [9], [13] or a related task such as prediction [14].”