2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning

By Diogo Carbonera Luvizon, David Picard, Hedi Tabia. 2018

535 citations recorded by the source corpus.

Action recognition and human pose estimation are closely related but both problems are generally handled as distinct tasks in the literature. In this work, we propose a multitask framework for jointly 2D and 3D pose estimation from still images and human action recognition from video sequences. We show that a single architecture can be used to solve the two problems in an efficient way and still achieves state-of-the-art results. Additionally, we demonstrate that optimization from end-to-end leads to significantly higher accuracy than separated learning. The proposed architecture can be trained with data from different categories simultaneously in a seamlessly way. The reported results on four datasets (MPII, Human3.6M, Penn Action and NTU) demonstrate the effectiveness of our method on the targeted tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Endocrinology, Diabetes and Metabolism, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Endocrinology, Diabetes and Metabolism

In 3D Vision & Pose

Citations in the corpus

NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “Speciﬁcally, many of them have been evaluated based on the preliminary version [47] of our dataset, or pre-trained on it for transfer learning for other tasks [43], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79].”
Feature Boosting Network For 3D Pose Estimation (2019, positive): “Recently, they have also been successfully applied to 3D pose estimation [5], [6], [8], [10], [11], [12], [13].”
Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “When additional pose data is available [37], the 3D joint positions are typically fed into a separate network.”
Convolutional Mesh Regression for Single-Image Human Shape Reconstruction (2019, background): “There is rich recent literature on 3D pose estimation in the form of a simplistic body skeleton, e.g., [3, 19, 22, 24, 25, 29, 30, 34, 35, 38, 40, 41, 42, 50, 51].”
BodyNet: Volumetric Inference of 3D Human Body Shapes (2018, unknown): “Recent work of [50] jointly learns 2D/3D pose”
MotionBERT: A Unified Perspective on Learning Human Motion Representations (2022, background): “The pioneering works [69, 108, 120] point out the inherent connection be-tween action recognition and human pose estimation.”
Deep 3D human pose estimation: A review (2021, unknown): “3D pose estimation can also be used to assist other computer vision tasks such as pose transfer (Li et al., 2019a), action recognition (Luvizon et al., 2018), human parsing (Xia et al., 2016), person image generation (Siarohin et al., 2018), animation (Weng et al., 2019), pose search (Ferrari et…”
3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training (2018, methodology): “(+) extra data – [50, 40, 58, 33] use 2D annotations from the MPII dataset, [40] uses additional data from the Leeds Sports Pose (LSP) dataset as well as ordinal annotations.”
Revisiting Skeleton-based Action Recognition (2021, methodology): “Other works [2,25,26,33,41] directly convert the coordinates in a skeleton sequence to a pseudo image with transformations, typically generate a 2D input of shapeK×T , whereK is the number of joints, T is the temporal length.”
Single-Shot Multi-person 3D Pose Estimation from Monocular RGB (2017, background): “Recent work approaches this more general 3D multi-person pose estimation problem by decomposing it into multiple single-person instances [31, 49], often with signiﬁcant redundancy in the decomposition [49].”
Exploiting Spatial-Temporal Relationships for 3D Pose Estimation via Graph Convolutional Networks (2019, background): “Despite the tremendous success achieved in recent years [8, 27, 28, 38, 44, 5, 16, 49, 25, 13], it remains a challenging problem due to the frequent selfocclusions and substantial depth ambiguity in 2D representations.”
Object Level Visual Reasoning in Videos (2018, background): “Pose estimation and activity recognition as a joint (multi-task) problem has recently shown to improve both tasks [25].”
MMTM: Multimodal Transfer Module for CNN Fusion (2019, background): “Multimodal Action Recognition in Videos Video [1, 14, 26] and skeleton [11, 27, 28] modalities have been extensively used for the action recognition task.”
Human Pose Regression by Combining Indirect Part Detection and Contextual Information (2017, unknown): “In what follows, we ﬁrst present the global architecture of our method, and then detail its most important parts.”
AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time (2022, background): “Several previous works have studied the softargmax operation to read continuous joint locations from heatmaps [41], [42], [43], [44], [45], [46].”
Monocular Total Capture: Posing Face, Body, and Hands in the Wild (2018, methodology): “Direct estimation methods predict 3D human pose directly from images, in the form of direct coordinate regression [46, 55, 56], voxel [42, 32, 66] or depth map [73].”
Skeleton-Based Action Recognition With Multi-Stream Adaptive Graph Convolutional Networks (2019, unknown): “8 RGB + 2D Pose [49] X X 85.”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “[147] learned a multi-task network to jointly handle 2D/3D pose estimation and action recognition from videos.”
GHUM & GHUML: Generative 3D Human Shape and Articulated Pose Models (2020, background): “Many image and video-based pose and shape estimation methods have also been proposed[33, 27, 25, 40, 22, 1, 12, 26, 34, 23].”
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction (2020, methodology): “a large-scale dataset with 11 subjects (7 with ground truth) and 3.6 million video frames in total. Each subject performs 15 actions and the human motion is recorded at 50 Hz. Following previous work [53,49,73,56], we adopt a 17-joint skeleton and train on ve subjects (S1, S5, S6, S7, S8) and test o”