Learning to Fuse 2D and 3D Image Cues for Monocular Body Pose Estimation

By Bugra Tekin, Pablo Márquez-Neila, Mathieu Salzmann, Pascal Fua. 2016

260 citations recorded by the source corpus.

Most recent approaches to monocular 3D human pose estimation rely on Deep Learning. They typically involve regressing from an image to either 3D joint coordinates directly or 2D joint locations from which 3D coordinates are inferred. Both approaches have their strengths and weaknesses and we therefore propose a novel architecture designed to deliver the best of both worlds by performing both simultaneously and fusing the information along the way. At the heart of our framework is a trainable fusion scheme that learns how to fuse the information optimally instead of being hand-designed. This yields significant improvements upon the state-of-the-art on standard 3D human pose estimation benchmarks.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Endocrinology, Diabetes and Metabolism

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Endocrinology, Diabetes and Metabolism

Citations in the corpus

End-to-End Recovery of Human Shape and Pose (2017, background): “[45], many recent methods estimate 3D joints directly from images in a deep learning framework [33, 43, 44, 51, 52].”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “Then, 2D-to-3D lifting is used to obtain 3D pose in the second stage, e.g., References [18, 111, 156, 168, 226, 307].”
Exploiting Temporal Information for 3D Human Pose Estimation (2017, unknown): “Tekin et al. [43] used a CNN to ﬁrst align bounding boxes of successive frames so that the person in the image is always at the center of the box and then extracted 3D HOG features densely over the spatio-temporal volume from which they regress the 3D pose of the central frame.”
Monocular human pose estimation: A survey of deep learning-based methods (2020, methodology): “, 2018a) (2) 2D-to-3D: (Martinez et al., 2017; Zhou et al., 2017; Tekin et al., 2017; Li and Lee, 2019; Qammaz and Argyros, 2019; Chen and Ramanan, 2017; Moreno-Noguer, 2017; Wang et al., 2018a; Yang et al., 2018)”
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions (2019, background): “A significant amount of research has focused on visual understanding of humans [8, 25, 36, 40, 59, 65, 70, 74] and objects [4, 28, 64, 66], in isolation from each other.”
Monocular 3D Pose and Shape Estimation of Multiple People in Natural Scenes: The Importance of Multiple Scene Constraints (2018, methodology): “Our work relates to recently developed deep architectures for 2d human pose estimation [4, 9, 21, 35, 36], 3d human pose estimation based on fitting volumetric models [2, 15], feedforward deep models for 3d prediction [18, 22, 40], as well as integrated deep models for 2d and 3d reasoning [23, 27, 3”
Learning to Estimate 3D Human Pose and Shape from a Single Color Image (2018, background): “Many recent works follow the end-to-end paradigm [48, 40, 42, 46, 55], using images as input to predict 3D joint locations [23, 45, 34, 28], regress 3D heatmaps [31], or classify the image in a particular pose class [39, 40].”
Ordinal Depth Supervision for 3D Human Pose Estimation (2018, methodology): “, [51, 64, 48, 25]) also use in-the-wild images with 2D keypoints for supervision, most of the improvement for our approach comes from augmenting training with ordinal depth relations for these examples.”
Semantic Graph Convolutional Networks for 3D Human Pose Regression (2019, methodology): “A couple of algorithms directly predicted 3D pose from the image [75], while others combined 2D heatmaps with volumetric representation [41], pairwise distance matrix estimation [36] or image cues [56] for 3D human pose regression.”
Convolutional Mesh Regression for Single-Image Human Shape Reconstruction (2019, background): “There is rich recent literature on 3D pose estimation in the form of a simplistic body skeleton, e.g., [3, 19, 22, 24, 25, 29, 30, 34, 35, 38, 40, 41, 42, 50, 51].”
Single-Shot Multi-person 3D Pose Estimation from Monocular RGB (2017, background): “[60] (70.81mm), and better than the recent results from Nie et al .”
Self-Supervised Learning of 3D Human Pose Using Multi-View Geometry (2019, methodology): “Single-view methods In many recent works, convolutional neural networks (CNN) are used to estimate the co-ordinates of the 3D joints directly from images [38, 39, 40, 35, 23].”
Hand Pose Estimation via Latent 2.5D Heatmap Regression (2018, methodology): “Instead of 2D keypoint locations, [48,49] use 2D heatmaps [26,27] as input and learn convolutional neural networks for 3D pose regression.”
Learning Monocular 3D Human Pose Estimation from Multi-view Images (2018, methodology): “Algorithms for 3D human pose estimation from single images [18, 27, 20, 15, 17, 23, 19, 31, 26, 25] have now matured to the point where they can operate in real-time and in the wild.”
LCR-Net++: Multi-Person 2D and 3D Pose Detection in Natural Images (2018, methodology): “Finally, some recent approaches treat 2D and 3D pose estimation jointly or iteratively [45], [46], [47], [48], [49].”
Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop (2019, background): “Recent works have made significant advances in the frontier of skeleton-based 3D human pose estimation from single images, with many approaches achieving impressive results [21, 23, 29, 33, 35, 45].”
Deep 3D human pose estimation: A review (2021, unknown): “…input images to 3D body joint positions can be categorized into two classes: detection-based methods (Pavlakos et al., 2017a; Luvizon et al., 2018) and regression-based methods (Li and Chan, 2014; Zhou et al., 2016a; Sun et al., 2017; Tekin et al., 2017; Zhou et al., 2017; Luvizon et al., 2019).”