Pose-conditioned Spatio-Temporal Attention for Human Action Recognition

By Fabien Baradel, Christian Wolf, Julien Mille. 2017

83 citations recorded by the source corpus.

We address human action recognition from multi-modal video data involving articulated pose and RGB frames and propose a two-stream approach. The pose stream is processed with a convolutional model taking as input a 3D tensor holding data from a sub-sequence. A specific joint ordering, which respects the topology of the human body, ensures that different convolutional layers correspond to meaningful levels of abstraction. The raw RGB stream is handled by a spatio-temporal soft-attention mechanism conditioned on features from the pose network. An LSTM network receives input from a set of image locations at each instant. A trainable glimpse sensor extracts features on a set of predefined locations specified by the pose stream, namely the 4 hands of the two people involved in the activity. Appearance features give important cues on hand motion and on objects held in each hand. We show that it is of high interest to shift the attention to different hands at different time steps depending on the activity itself. Finally a temporal attention mechanism learns how to fuse LSTM features over time. We evaluate the method on 3 datasets. State-of-the-art results are achieved on the largest dataset for human activity recognition, namely NTU-RGB+D, as well as on the SBU Kinect Interaction dataset. Performance close to state-of-the-art is achieved on the smaller MSR Daily Activity 3D dataset.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Jun Liu
2. G. Wang
3. Hossein Rahmani
4. Amir Shahroudy
5. Alex ChiChung Kot
6. Ling-yu Duan
7. Qiuhong Ke
8. Zehua Sun

In Human-Computer Interaction

1. Jun Liu
2. Hossein Rahmani
3. G. Wang
4. Amir Shahroudy
5. Alex ChiChung Kot
6. Ling-yu Duan
7. Qiuhong Ke
8. Zehua Sun

Citations in the corpus

• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, methodology): “Specifically, many of them have been evaluated based on the preliminary version [47] of our dataset, or pre-trained on it for transfer learning for other tasks [43], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79].”
• 2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning (2018, methodology): “If we consider only RGB frames as input, our method improves over [3] by 9.9%.”
• Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN (2018, methodology): “CS CV Deep learning on Lie Group [15] 61.37% 66.95% JTM+CNN [49] 73.40% 75.20% Res-TCN [22] 74.30% 83.10% SkeletonNet(CNN) [20] 75.94% 81.16% JDM+CNN [31] 76.20% 82.30% Clips+CNN+MTLN [21] 79.57% 84.83% Enhanced Visualization+CNN [33] 80.03% 87.21% 1 Layer RNN [46] 56.02% 60.24% 2 Layer RNN [46] 56.”
• Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “The two-stream architecture was also investigated in [286], [288], where the classification scores from a CNN model [288] or an RNN model [286] trained on skeleton data and a skeleton conditioned spatio-temporal attention network trained on RGB data were fused for final classification.”