Recognizing Human Actions as the Evolution of Pose Estimation Maps

By Mengyuan Liu, Junsong Yuan. 2018

322 citations recorded by the source corpus.

Most video-based action recognition approaches choose to extract features from the whole video to recognize actions. The cluttered background and non-action motions limit the performances of these methods, since they lack the explicit modeling of human body movements. With recent advances of human pose estimation, this work presents a novel method to recognize human action as the evolution of pose estimation maps. Instead of relying on the inaccurate human poses estimated from videos, we observe that pose estimation maps, the byproduct of pose estimation, preserve richer cues of human body to benefit action recognition. Specifically, the evolution of pose estimation maps can be decomposed as an evolution of heatmaps, e.g., probabilistic maps, and an evolution of estimated 2D human poses, which denote the changes of body shape and body pose, respectively. Considering the sparse property of heatmap, we develop spatial rank pooling to aggregate the evolution of heatmaps as a body shape evolution image. As body shape evolution image does not differentiate body parts, we design body guided sampling to aggregate the evolution of poses as a body pose evolution image. The complementary properties between both types of images are explored by deep convolutional neural networks to predict action label. Experiments on NTU RGB+D, UTD-MHAD and PennAction datasets verify the effectiveness of our method, which outperforms most state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Liu
2. G. Wang
3. Alex ChiChung Kot
4. Ling-yu Duan
5. Amir Shahroudy
6. Pichao Wang
7. Hanqing Lu
8. Jian Cheng

In Computer Vision and Pattern Recognition

1. Jun Liu
2. G. Wang
3. Hanqing Lu
4. Alex ChiChung Kot
5. Jian Cheng
6. Liang Wang
7. Amir Shahroudy
8. Ling-yu Duan

Citations in the corpus

• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, methodology): “The methods in [30], [47], [48], [94], [102], [105] all use RNN/LSTM for 3D action recognition, and the approaches in [49], [103], [104], [106], [107] all use convolutional networks for spatial and temporal modeling.”
• MMTM: Multimodal Transfer Module for CNN Fusion (2019, methodology): “Multimodal Action Recognition in Videos Video [1, 14, 26] and skeleton [11, 27, 28] modalities have been extensively used for the action recognition task.”
• Decoupling GCN with DropGraph Module for Skeleton-Based Action Recognition (2020, background): “Compared with action recognition with RGB video, skeleton-based action recognition is robust to circumstance changes and illumination variations [3,40,36,48,50,25,46,17,33,34,29].”
• Skeleton-Based Action Recognition With Multi-Stream Adaptive Graph Convolutional Networks (2019, unknown): “s the previous methods with a large margin. Methods Pose RGB CS (%) CV (%) DSSCA-SSLM [47] X X 74.9 - Chained Network [48] X X 80.8 - RGB + 2D Pose [49] X X 85.5 - Glimpse Clouds [50] X 86.6 93.2 PEM [51] X X 91.2 95.3 I3D+RNN+Attention [52] X X 93.0 95.4 MS-AAGCN X 90.0 96.2 RNX3D101 X 85.3 92.6 RN”
• Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition (2020, unknown): “bject C-Setup Year Dynamic Skeleton [7] 50.8 54.7 2015 PA-LSTM [23] 25.5 26.3 2016 ST-LSTM+TS [18] 55.7 57.9 2016 FSNet [17] 59.9 62.4 2019 2s-ALSTM [19] 61.2 63.3 2017 MT CNN [9] 62.2 61.8 2018 BPEM [20] 64.6 66.9 2018 2s-AGCN [26] 82.9 84.9 2019 MS-G3D Net [21] 86.9 88.4 2020 Dynamic GCN (ours) 87”
• Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition (2019, background): “8 Body Pose Evolution Map [31] 2018 64.”
• Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition (2020, background): “Table 4 compares non-graph [26, 27, 16, 28] and graph-based methods [33].”
• Spatial Temporal Transformer Network for Skeleton-based Action Recognition (2020, unknown): “Map [9] 64.”
• MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation (2021, background): “3D human pose estimation (HPE) from monocular videos is a fundamental vision task with a wide range of applications, such as action recognition [22, 23, 39], humancomputer interaction [7], and augmented/virtual reality [30].”