Learning Monocular 3D Human Pose Estimation from Multi-view Images

By Helge Rhodin, Jörg Spörri, Isinsu Katircioglu, Victor Constantin, Frederick Meyer, Erich Müller, Mathieu Salzmann, Pascal Fua. 2018

258 citations recorded by the source corpus.

Accurate 3D human pose estimation from single images is possible with sophisticated deep-net architectures that have been trained on very large datasets. However, this still leaves open the problem of capturing motions for which no such database exists. Manual annotation is tedious, slow, and error-prone. In this paper, we propose to replace most of the annotations by the use of multiple views, at training time only. Specifically, we train the system to predict the same pose in all views. Such a consistency constraint is necessary but not sufficient to predict accurate poses. We therefore complement it with a supervised loss aiming to predict the correct pose in a small set of labeled images, and with a regularization term that penalizes drift from initial predictions. Furthermore, we propose a method to estimate camera pose jointly with human pose, which lets us utilize multiview footage where calibration is difficult, e.g., for pan-tilt or moving handheld cameras. We demonstrate the effectiveness of our approach on established benchmarks, as well as on a new Ski dataset with rotating cameras and expert ski motion, for which annotations are truly hard to obtain.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learnable Triangulation of Human Pose (2019, background): “, namely [16]. Multi-view view 3D pose estimation. Studies of multiview 3D human pose estimation are generally aimed at getting the ground-truth annotations for the monocular 3D human pose estimation [14,5]. The work [6] proposed concatenating joints’ 2D coordinates from all views into a single batc”
Self-Supervised Learning of 3D Human Pose Using Multi-View Geometry (2019, methodology): “[31] utilize multi-view consistency constraints to supervise a network.”
Disentangling Latent Hands for Image Synthesis and Pose Estimation (2018, background): “In particular, for the monocular scenario, one possibility would be to collect multi-view [23] and also multi-modal data, i.”
Recovering Accurate 3D Human Pose in the Wild Using IMUs and a Moving Camera (2018, background): “Several works lift 2D detections to 3D using learning or geometric reasoning [18, 29, 35, 9, 26, 49, 33, 48, 44, 34, 13, 43, 45].”
C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion (2019, methodology): “Besides the fully supervised methods [26, 27], several works have explored multi-view supervision [21, 30, 32], ordinal depth supervision [29], unpaired 2D-3D data [31, 36, 41, ?] or videos [?] to alleviate the need for full 2D-3D annotations.”
Deep 3D human pose estimation: A review (2021, unknown): “Typical meth-ods include fusing multi-view 2D heatmaps (Pavlakos et al., 2017b; Tome et al., 2018; Qiu et al., 2019), enforcing multiple view consistency (Rhodin et al., 2018a,b), triangulation (Kocabas et al., 2019; Iskakov et al., 2019), and utilizing the SMPL model (Liang and Lin, 2019).”
Monocular human pose estimation: A survey of deep learning-based methods (2020, methodology): “6M is calculated after the alignment of predictions and ground truth with a rigid transformation using Procrustes Analysis (Gower, 1975), which is also called reconstruction error (Kanazawa et al., 2018; Pavlakos et al., 2018b), P-MPJPE (Rhodin et al., 2018a) or PA-MPJPE (Sun et al., 2018).”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, background): “…and performance capture Per-ceiving humans from monocular RGB data [12, 29, 31, 41, 43, 44, 58, 59, 64, 87] and under multiple views [37–40 , 62] Following the success of pixel-aligned implicit function learning [64, 65], recent methods can capture human performance from sparse [38, 80] or…”
Deep learning tools for the measurement of animal behavior in neuroscience (2019, background): “Even though very large data sets have been amassed to build robust DNNs (19, 30–32), they can still fail in sufﬁciently different scenarios (50, 92, 93).”
Expressive Body Capture: 3D Hands, Face, and Body From a Single Image (2019, unknown): “3D models [7, 20, 24, 30, 35, 46, 54, 66, 72].”
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction (2020, unknown): “Starting from multi-view setups at training time, [34, 35] propose weak supervision strategies for monocular human pose estimation.”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “NMPJPE is deined as the MPJPE after normalizing the predicted positions in scale to the reference [207].”