Synthesizing Training Images for Boosting Human 3D Pose Estimation

By Wenzheng Chen, Huan Wang, Yangyan Li, Hao Su, Zhenhua Wang, Changhe Tu, Dani Lischinski, Daniel Cohen-Or, Baoquan Chen. 2016

299 citations recorded by the source corpus.

Human 3D pose estimation from a single image is a challenging task with numerous applications. Convolutional Neural Networks (CNNs) have recently achieved superior performance on the task of 2D pose estimation from a single image, by training on images with 2D annotations collected by crowd sourcing. This suggests that similar success could be achieved for direct estimation of 3D poses. However, 3D poses are much harder to annotate, and the lack of suitable annotated training images hinders attempts towards end-to-end solutions. To address this issue, we opt to automatically synthesize training images with ground truth pose annotations. Our work is a systematic study along this road. We find that pose space coverage and texture diversity are the key ingredients for the effectiveness of synthetic training data. We present a fully automatic, scalable approach that samples the human pose space for guiding the synthesis procedure and extracts clothing textures from real images. Furthermore, we explore domain adaptation for bridging the gap between our synthetic training images and real testing photos. We demonstrate that CNNs trained with our synthetic images out-perform those trained with real photos on 3D pose estimation tasks.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

DensePose: Dense Human Pose Estimation in the Wild (2018, background): “Synthetic ground-truth can be established by rendering images using surface-based models [33, 32, 37, 11, 5, 30].”
Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection (2017, background): “Similarly, [5, 50] render 3D humans in scenes and use this data for pose estimation.”
FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation (2016, background): “…shown that the domain discrepancy distance may be minimized through an adversarial learning procedure, whereby simultaneously a domain classiﬁer is trained to best distinguish the source and target distributions and the representation space is updated according to the inverse objec-tive [32, 2, 7].”
Monocular 3D Human Pose Estimation in the Wild Using Improved CNN Supervision (2016, methodology): “6m [27] are recorded indoors and test on similar looking scenes as the training set, the Human3D+ [15] test set was recorded with sensor suits that inﬂuence appearance and lacks global alignment, and the MARCoNI set [17] is mark-erless through manual annotation, but shows mostly walking motions and ”
Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars (2017, background): “…et al. 2013; Shakhnarovich et al. 2003; Yasin et al. 2016; Du et al. 2016; Ghezelghieh et al. 2016; Rogez and Schmid 2016; Zhou et al. 2016; Chen et al. 2016; Varol et al. 2017), depth prediction (Su et al. 2014), pedestrian detection (Marin et al. 2010; Pishchulin et al. 2011; Vázquez et…”
Learning to Fuse 2D and 3D Image Cues for Monocular Body Pose Estimation (2016, methodology): “2, we report the results of two instances of our model: one trained on the standard KTH training data, and one pretrained on the synthetic 3D human pose dataset of [12] and fine-tuned on the KTH dataset.”
Fusing 2D Uncertainty and 3D Cues for Monocular Body Pose Estimation (2016, methodology): “Since the training set is quite small, we propose to pretrain our network on the recently released synthetic dataset [11], which contains images of sports players with their corresponding 3D poses.”
Single-Shot Multi-person 3D Pose Estimation from Monocular RGB (2017, background): “There is work on generating synthetic images [48, 10] from mocap data, however the resulting images are not plausible.”
Semantic Graph Convolutional Networks for 3D Human Pose Regression (2019, methodology): “Other methods use synthetic datasets which are generated from deforming a human template model with the ground truth [8, 42, 48] or introduce loss functions involving high-level knowledge [40, 53, 69] in addition to joints.”
Monocular 3D Human Pose Estimation Using Transfer Learning and Improved CNN Supervision (2016, methodology): “We cover a wide range of viewpoints, with five cameras mounted at chest height with a roughly 15◦ elevation variation similar to the camera orientation jitter in other datasets [17].”
Geometric Image Synthesis (2018, background): “GAN-based renement [25]. More variability can be obtained when rendering entire 3D CAD models into the image. Several works consider the augmentation of images with virtual [26,27] or scanned humans [28,29]. In contrast, Alhaija et al. [6,9] consider the problem of augmenting scenes from the KITTI d”
Generative adversarial networks: introduction and outlook (2017, background): “There are researchers [26], [27] using the adversarial idea to train neural network with domain adaptation: the feature generator transforms the source-domain data and the target-domain data into high-level abstract features, making the domains of features difﬁcult to discriminate as far as…”
Ordinal Depth Supervision for 3D Human Pose Estimation (2018, methodology): “The most straightforward solution is to use graphics to augment the training data [13, 53, 26].”
Towards 3D Human Pose Estimation in the Wild: A Weakly-Supervised Approach (2017, background): “An alternative approach for 3D human pose estimation is to train from synthetic datasets which are generated from deforming a human template model with known 3D ground truth [6, 22].”
Lucid Data Dreaming for Object Tracking (2017, unknown): “Synthetic renderings [37], video game environment [45], mix-synthetic and real images [56, 9, 12] have shown promise, but require task-appropriate 3d models.”
Toward Characteristic-Preserving Image-based Virtual Try-On Network (2018, methodology): “[4] used a SCAPE [1] body model to generate synthetic people.”
LCR-Net: Localization-Classification-Regression for Human Pose (2017, background): “synthesis that shows promising results in the wild [5, 23].”