Synthesizing Images of Humans in Unseen Poses

By Guha Balakrishnan, Amy Zhao, Adrian V. Dalca, Frédo Durand, John V. Guttag. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

327 citations recorded by the source corpus.

We address the computational problem of novel human pose synthesis. Given an image of a person and a desired pose, we produce a depiction of that person in that pose, retaining the appearance of both the person and background. We present a modular generative neural network that synthesizes unseen poses using training pairs of images and poses taken from human action videos. Our network separates a scene into different body part and background layers, moves body parts to new locations and refines their appearances, and composites the new foreground with a hole-filled background. These subtasks, implemented with separate modules, are trained jointly using only a single target image as a supervised label. We use an adversarial discriminator to force our network to synthesize realistic details conditioned on pose. We demonstrate image synthesis results on three action classes: golf, yoga/workouts and tennis, and show that our method produces accurate results within action classes as well as across action classes. Given a sequence of desired poses, we also produce coherent videos of actions.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1bd2ce1d6b4f675dfba3a12ca0254c1ba8f6689a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Animating Arbitrary Objects via Deep Motion Transfer (2018, background): “However, standard convolutional encoder-decoders are not designed to handle large pixel-to-pixel misalignment between the input and output images [31, 3, 14].”
The Creation and Detection of Deepfakes (2020, methodology): “In [14], the authors break the process down into three stages, trained end-to-end: (1) use a U-Net to segment xt ’s body parts and then orient them according to the source pose ps , (2) use a second U-Net to generate the body xд from the composite, and (3) use a third U-Net to perform in-painting on”
NeuMan: Neural Human Radiance Field from a Single Video (2022, unknown): “Majority of the literature [1,22,27,20,35] only consider the problem of reposing a human from a source image to a target image without changing the viewing angle which is essential for enabling new immersive experiences.”
Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video (2020, background): “Several approaches address related problems to ours, such as generating images of humans in new poses [2, 41, 49, 64] or body reenactment from monocular videos [8].”
Unsupervised Discovery of Parts, Structure, and Dynamics (2019, methodology): “Another line of related work is on future state prediction in either image pixels (Xue et al., 2016; Mathieu et al., 2016; Lotter et al., 2017; Lee et al., 2018; Balakrishnan et al., 2018b) or object trajectories (Kitani et al., 2017; Walker et al., 2016).”
First Order Motion Model for Image Animation (2020, background): “Unfortunately, these pre-trained models are built using costly ground-truth data annotations [2, 27, 31] and are not available in general for an arbitrary object category.”
Single-Shot Freestyle Dance Reenactment (2020, background): “Pose Warping[1] generates a new frame by transforming each body part of the target, based on pose key- points of the source and target images, followed by a fusion operation.”
Everybody Dance Now (2018, methodology): “Videos of resolution 144×256 (as this is the highest resolution that PoseWarp baseline can produce) were shown, and after each pair, participants were given Region Metric FBF FBF+TS FBF+TS+FG Fa ce SSIM 0.784 0.811 0.816 LPIPS 0.045 0.039 0.036 B od y SSIM 0.828 0.838 0.838 LPIPS 0.057 0.051 0.050 (”
Progressive Pose Attention Transfer for Person Image Generation (2019, background): “[2] presented a GAN network that decomposes the person image generation task into foreground and background generation, and then combines to form the final image.”
Geometry-Contrastive GAN for Facial Expression Transfer. (2018, background): “Some researchers have attempted to incorporate continuous information such as geometry into cGANs [2, 13, 17, 22, 27].”
ClothFlow: A Flow-Based Model for Clothed Person Generation (2019, background): “Besides, several recent works estimate the transformations in a non-learnable fashion [2, 14, 36]— they use keypoints or clothing masks to compute the parameters of a transformation, which is unaware of the appearance correspondence and can only roughly align textures.”
Few-shot Video-to-Video Synthesis (2019, background): “To achieve high quality generation results, existing human pose transfer methods largely utilize human body priors such as body part modeling [1] or human surface-based coordinate mapping [36].”
MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generation (2019, background): “Others require strong supervision in the form of keypoint/pose/mask annotations [31, 1, 27, 9], which limits their scalability, and still fall short of disentangling all of the four factors outlined above.”
HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video (2022, background): “Other than free-viewpoint rendering, there is another related active research field that focus on human motion retargeting either in 2D [2, 7, 35, 43, 54, 67, 68] or 3D [19, 20, 25, 32, 49, 53, 69, 74].”