MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild

By Grégory Rogez, Cordelia Schmid. 2016

282 citations recorded by the source corpus.

This paper addresses the problem of 3D human pose estimation in the wild. A significant challenge is the lack of training data, i.e., 2D images of humans annotated with 3D poses. Such data is necessary to train state-of-the-art CNN architectures. Here, we propose a solution to generate a large set of photorealistic synthetic images of humans with 3D pose annotations. We introduce an image-based synthesis engine that artificially augments a dataset of real images with 2D human pose annotations using 3D Motion Capture (MoCap) data. Given a candidate 3D pose our algorithm selects for each joint an image whose 2D pose locally matches the projected 3D pose. The selected images are then combined to generate a new synthetic image by stitching local image patches in a kinematically constrained manner. The resulting images are used to train an end-to-end CNN for full-body 3D pose estimation. We cluster the training data into a large number of pose classes and tackle pose estimation as a K-way classification problem. Such an approach is viable only with large training sets such as ours. Our method outperforms the state of the art in terms of 3D pose estimation in controlled environments (Human3.6M) and shows promising results for in-the-wild images (LSP). This demonstrates that CNNs trained on artificial images generalize well to real images.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

DensePose: Dense Human Pose Estimation in the Wild (2018, background): “Synthetic ground-truth can be established by rendering images using surface-based models [33, 32, 37, 11, 5, 30].”
Learning from Simulated and Unsupervised Images through Adversarial Training (2016, background): “various prediction tasks, including gaze estimation [43], text detection and classification in RGB images [9, 15], font recognition [42], object detection [10, 27], hand pose estimation in depth images [38, 37], scene recognition in RGB-D [11], semantic segmentation of urban scenes [31], and human p”
End-to-End Recovery of Human Shape and Pose (2017, methodology): “Rogez and Schmid [35] artificially endow 3D annotations to images with 2D pose annotation using MoCap data.”
Data Augmentation in Emotion Classification Using Generative Adversarial Networks (2017, background): “In the ﬁeld of 3D motion capture, 2D images are used for generating 3D ones [24].”
Sim2real transfer learning for 3D human pose estimation: motion to the rescue (2019, background): “For instance, some work constructs 3D datasets entirely by stitching together 2D images [67]; other works use feature selection [51] and stage-wise training [16] to improve transfer.”
Improving Deep Learning using Generic Data Augmentation (2017, background): “This scheme has become increasingly popular [Rogez and Schmid, 2015; Peng et al., 2015] as it has the ability to generate richer training data compared to the generic augmentation methods.”
Frankenstein: Learning Deep Face Representations Using Small Data (2016, unknown): “datasets has been explored by a number of authors [20], [33], [38], [39].”
Exploiting Temporal Information for 3D Human Pose Estimation (2017, background): “Rogesz and Schmid [39] and Varol et al. [38] augmented the training data with synthetic images and trained CNNs to predict 3D poses from real images.”
Integral Human Pose Regression (2017, methodology): “om the quantization problem. 2.1 Joint 3D and 2D training A lack of diverse training data is a severe problem for 3D human pose estimation. Several eorts have been made to combine 3D and 2D training [55,31,43,51,41]. Since integral regression provides a unied setting for both 2D and 3D pose estimati”
Single-Shot Multi-person 3D Pose Estimation from Monocular RGB (2017, background): “There is work on generating synthetic images [48, 10] from mocap data, however the resulting images are not plausible.”
A Simple Yet Effective Baseline for 3d Human Pose Estimation (2017, background): “Recently, some systems have explored the possibility of directly inferring 3d poses from images with end-to-end deep architectures [33, 45], and other systems argue that 3d reasoning from colour images can be achieved by training on synthetic data [38, 48].”
Semantic Graph Convolutional Networks for 3D Human Pose Regression (2019, methodology): “Other methods use synthetic datasets which are generated from deforming a human template model with the ground truth [8, 42, 48] or introduce loss functions involving high-level knowledge [40, 53, 69] in addition to joints.”
Learning to Fuse 2D and 3D Image Cues for Monocular Body Pose Estimation (2016, methodology): “[77] - - - Rogez & Schmid [53] - - - Tekin et al.”
LCR-Net: Localization-Classification-Regression for Human Pose (2017, methodology): “in regression [14, 31] and classification [23] approaches.”
Monocular 3D Human Pose Estimation Using Transfer Learning and Improved CNN Supervision (2016, background): “Rogez and Schmid [58] stitch multiple real 2D images to match projected 3D poses to increase variety, and Pishchulin et al. warp human size in images with a parametric body model[52].”
Self-supervised Learning of Motion Capture (2017, background): “Rogez and Schmid [29] obtain large scale RGB to 3D pose synthetic annotations by rendering synthetic 3D human models against realistic backgrounds [29], a dataset also used in this work.”
LCR-Net++: Multi-Person 2D and 3D Pose Detection in Natural Images (2018, methodology): “Recently, this has been naturally extended to end-to-end mappings using CNN architectures, either in monocular images [7], [8], [12], [41], [42], [43] or in videos [11], [44].”
Towards 3D Human Pose Estimation in the Wild: A Weakly-Supervised Approach (2017, background): “An alternative approach for 3D human pose estimation is to train from synthetic datasets which are generated from deforming a human template model with known 3D ground truth [6, 22].”
Continual Adaptation of Visual Representations via Domain Randomization and Meta-learning (2020, background): “The use of data augmentation has a long history in computer vision [55, 57, 9, 8, 30, 51].”