Learning from Synthetic Humans

By Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, Cordelia Schmid. 2017

1,053 citations recorded by the source corpus.

Estimating human pose, shape, and motion from images and videos are fundamental challenges with many applications. Recent advances in 2D human pose estimation use large amounts of manually-labeled training data for learning convolutional neural networks (CNNs). Such data is time consuming to acquire and difficult to extend. Moreover, manual labeling of 3D pose, depth and motion is impractical. In this work we present SURREAL (Synthetic hUmans foR REAL tasks): a new large-scale dataset with synthetically-generated but realistic images of people rendered from 3D sequences of human motion capture data. We generate more than 6 million frames together with ground truth pose, depth maps, and segmentation masks. We show that CNNs trained on our synthetic dataset allow for accurate human depth estimation and human part segmentation in real RGB images. Our results and the new dataset open up new possibilities for advancing person analysis using cheap and large-scale synthetic data.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Social Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Social Psychology

Citations in the corpus

DensePose: Dense Human Pose Estimation in the Wild (2018, methodology): “This has recently been applied to human pose in the SURREAL dataset of [45], where the SMPL model [28] was rendered with the CMU Mocap dataset poses [29].”
End-to-End Recovery of Human Shape and Pose (2017, methodology): “We thank Naureen Mahmood for providing MoShed datasets and mesh retargeting for character animation, Dushyant Mehta for his assistance on MPIINF-3DHP, and Shubham Tulsiani, Abhishek Kar, Saurabh Gupta, David Fouhey and Ziwei Liu for helpful discussions.”
Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection (2017, background): “Similarly, [5, 50] render 3D humans in scenes and use this data for pose estimation.”
Learning to Reconstruct People in Clothing From a Single RGB Camera (2019, methodology): “Variants of random forests and neural networks have been used [24, 23, 25, 75] to regress shape from silhouettes.”
PoseTrack: A Benchmark for Human Pose Estimation and Tracking (2017, methodology): “We envision that future work will combine manually labeled data with other techniques such as transfer learning from other datasets such as [5], inferring sequences of poses by propagating annotations from reliable keyframes [6], and leveraging synthetic training data as in [47].”
GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB (2017, methodology): “By using the background masks from the original synthetic images, we perform background augmentation by compositing GANerated images (foreground) with random images (background) [53, 19, 28].”
Monocular, One-stage, Regression of Multiple 3D People (2020, background): “Recently, great progress has been made in monocular 3D human pose and shape estimation, particularly in images with a single person [20, 22, 23, 48, 51].”
3D-RCNN: Instance-Level 3D Object Reconstruction via Render-and-Compare (2018, methodology): “Note that [52] trains two distinct conv-nets specific to the task of depth prediction and body parts segmentation.”
Learning Monocular 3D Human Pose Estimation from Multi-view Images (2018, background): “Several authors have proposed to leverage recent advances in computer graphics and human rendering [14] to rely on fully-synthetic images [1, 29].”
Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars (2017, background): “…Shakhnarovich et al. 2003; Yasin et al. 2016; Du et al. 2016; Ghezelghieh et al. 2016; Rogez and Schmid 2016; Zhou et al. 2016; Chen et al. 2016; Varol et al. 2017), depth prediction (Su et al. 2014), pedestrian detection (Marin et al. 2010; Pishchulin et al. 2011; Vázquez et al. 2014; Hattori…”
Soccer on Your Tabletop (2018, methodology): “The method of [49] assigned a large number of foreground pixels to the background.”
Learning Joint Reconstruction of Hands and Manipulated Objects (2019, positive): “Body poses and shapes are varied by sampling from the same distribution as in SURREAL [72]; i.e., sampling poses from the CMU MoCap database [1] and shapes from CAESAR [50].”
GRAB: A Dataset of Whole-Body Human Grasping of Objects (2020, background): “However, GRAB can support image-based inference [8, 28] by enabling rendering of synthetic human-object interaction [24, 53, 70] or learning priors to regularizing ill-posed inference of human-object interaction from 2D images [42].”
Adversarial Geometry and Lighting using a Differentiable Renderer (2018, background): “Many previous works use rendered images during training, as rendering provides a theoretically inﬁnite supply of input data [29, 5, 43, 39, 33, 16].”
LCR-Net++: Multi-Person 2D and 3D Pose Detection in Natural Images (2018, methodology): “In (b), we show a synthetic “surreal” [54] image, i.”
3D-CODED: 3D Correspondences by Deep Deformation (2018, methodology): “Furthermore, while our network can take advantage of known correspondences between the template and the example shapes, which are typically available when they have been generated using some parametric model [6,45], we show it can also be trained without correspondence supervision.”
PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization (2019, background): “To efficiently render the digital humans, Lambertian diffuse shading with surface normal and spherical harmonics are typically used due to its simplicity and efficiency [16, 11].”
Conditional Generative Adversarial Network for Structured Domain Adaptation (2018, background): “Recent advances in computer graphics have stimulated the interests to train deep learning models on photo-realistic synthetic data with computergenerated annotations, and apply them in real-world settings [12, 29, 47, 37, 41, 48, 20, 19, 32, 22, 34, 33].”
Fake it till you make it: face analysis in the wild using synthetic data alone (2021, methodology): “The computer vision community has used synthetic data for many tasks, including object recognition [23, 44, 51, 73], scene understanding [12, 25, 47, 50], eye tracking [63, 68], hand tracking [40, 61], and full-body analysis [41, 59, 65].”