Self-Supervised Learning of 3D Human Pose Using Multi-View Geometry

By Muhammed Kocabas, Salih Karagoz, Emre Akbas. Computer Vision and Pattern Recognition, 2019

298 citations recorded by the source corpus.

Training accurate 3D human pose estimators requires large amount of 3D ground-truth data which is costly to collect. Various weakly or self supervised pose estimation methods have been proposed due to lack of 3D data. Nevertheless, these methods, in addition to 2D ground-truth poses, require either additional supervision in various forms (e.g. unpaired 3D ground truth data, a small subset of labels) or the camera parameters in multiview settings. To address these problems, we present EpipolarPose, a self-supervised learning method for 3D human pose estimation, which does not need any 3D ground-truth data or camera extrinsics. During training, EpipolarPose estimates 2D poses from multi-view images, and then, utilizes epipolar geometry to obtain a 3D pose and camera geometry which are subsequently used to train a 3D pose estimator. We demonstrate the effectiveness of our approach on standard benchmark datasets (i.e. Human3.6M and MPI-INF-3DHP) where we set the new state-of-the-art among weakly/self-supervised methods. Furthermore, we propose a new performance measure Pose Structure Score (PSS) which is a scale invariant, structure aware measure to evaluate the structural plausibility of a pose with respect to its ground truth. Code and pretrained models are available at https://github.com/mkocabas/EpipolarPose

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/6d4d26a0444d8047b2aaa867ebef338345be9732

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion (2019, methodology): “Unlike other weakly supervised methods [10, 20], Pose-GAN does not assume knowl-”
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting (2021, unknown): “Apart from traditional object classification, detection or semantic segmentation, self-supervision has been extended to sub-domains like human pose-estimation [32], co-part segmentation [27], and depth estimation [19].”
Deep 3D human pose estimation: A review (2021, unknown): “Typical meth-ods include fusing multi-view 2D heatmaps (Pavlakos et al., 2017b; Tome et al., 2018; Qiu et al., 2019), enforcing multiple view consistency (Rhodin et al., 2018a,b), triangulation (Kocabas et al., 2019; Iskakov et al., 2019), and utilizing the SMPL model (Liang and Lin, 2019).”
Exploiting Temporal Contexts With Strided Transformer for 3D Human Pose Estimation (2021, background): “The mean per joint position error (MPJPE) is the average Euclidean distance between the ground truth and predicted positions of the joints, which is referred to as protocol #1 in many works [41], [58].”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “[101] employed epipolar geometry to match paired multi-view poses for 3D pose reconstruction and generalized their methods to new multi-view camera environments.”
Self-Supervised ECG Representation Learning for Emotion Recognition (2020, methodology): “In another work in [33], 3 D pose estimation was tackled using self-supervised learning.”