Joint action recognition and pose estimation from video

By Xiaohan Nie, Caiming Xiong, Song-Chun Zhu. 2015

207 citations recorded by the source corpus.

Action recognition and pose estimation from video are closely related tasks for understanding human motion, most methods, however, learn separate models and combine them sequentially. In this paper, we propose a framework to integrate training and testing of the two tasks. A spatial-temporal And-Or graph model is introduced to represent action at three scales. Specifically the action is decomposed into poses which are further divided to mid-level ST-parts and then parts. The hierarchical structure of our model captures the geometric and appearance variations of pose at each frame and lateral connections between ST-parts at adjacent frames capture the action-specific motion information. The model parameters for three scales are learned discriminatively, and action labels and poses are efficiently inferred by dynamic programming. Experiments demonstrate that our approach achieves state-of-art accuracy in action recognition while also improving pose estimation.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Space-Time Representation of People Based on 3D Skeletal Data: A Review (2016, methodology): “[143] ST Parts BoW Body Dict X X Anirudh et al.”
2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning (2018, background): “Previous approaches have explored classical methods for features extraction [55, 23], where the key idea is to use body joint locations to select visual features in space and time.”
A Hierarchical Pose-Based Approach to Complex Action Understanding Using Dictionaries of Actionlets and Motion Poselets (2016, methodology): “Some methods have tackled the problem of jointly recognizing actions and poses in videos [20] and still images [37], with the hope to create positive feedback by solving both tasks simultaneously.”
Action recognition in RGB-D egocentric videos (2017, background): “Action recognition [1–5] is a broadly researched field, which attempts to discriminate the action category in a video.”
Predicting the Where and What of Actors and Actions through Online Action Localization (2016, unknown): “[37].”
PoTion: Pose MoTion Representation for Action Recognition (2018, unknown): “[44] use a similar strategy leveraging a hierarchy of human body parts.”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “In skeleton-based HAR, some of the existing methods [179], [180] focused on extracting hand-crafted spatial and temporal features from the skeleton sequence for classification.”
You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions (2019, background): “Third-person body pose and interactions There is extensive literature on human body pose estimation from the traditional third-person viewpoint, where the person is entirely visible [53, 50, 64, 13].”
Pose for Action - Action for Pose (2016, methodology): “13], [19], [26], [37]. However, utilizing action recognition to aid human pose estimation is not well studied, in particular not in the context of 2D human pose estimation. There are only a few works [15], [31], [39], [40], [42] that showed the beneﬁt of it. The approaches in [39], [42] rely on stro”
Chained Predictions Using Convolutional Neural Networks (2016, methodology): “We compare to previous work on the Penn Action dataset [41].”
Dynamic Kernel Distillation for Efficient Pose Estimation in Videos (2019, methodology): “Prior CNN based methods to solve this task [10, 22, 25, 20] usually use a large network to extract representative features for every frame and localize body joints based on them via pixel-wise classification.”
Monocular 3D Human Pose Estimation by Predicting Depth on Joints (2017, methodology): “Inspired by those graphical model based pose estimation methods [38, 24, 20, 18], we represent human pose as a tree structure based on the kinematic relation of skeleton.”
Simultaneous Feature and Body-Part Learning for real-time robot awareness of human behaviors (2017, methodology): “A spatial-temporal And-Or graph model was implemented in [18] to represent humans at three levels including poses, spatiotemporal-parts, and parts.”
Recognizing Human Actions as the Evolution of Pose Estimation Maps (2018, methodology): “We follow [48] to split the data into half and half for training and testing.”
A Review of Human Activity Recognition Methods (2015, background): “Nie et al. (2015) also divided human pose into smaller mid-level spatiotemporal parts.”
Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video (2015, background): “, [50, 4, 49], and more recently using deep learning, e.”
MonoCap: Monocular Human Motion Capture using a CNN Coupled with a Geometric Prior (2017, unknown): “, [7], [8], [9], and more recently using deep learning, e.”
RPAN: An End-to-End Recurrent Pose-Attention Network for Action Recognition in Videos (2017, methodology): “Alternatively, human poses have proven useful for action recognition [14, 15, 17, 25, 40].”