Pose Guided Person Image Generation

By Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, Luc Van Gool. 2017

898 citations recorded by the source corpus.

This paper proposes the novel Pose Guided Person Generation Network (PG$^2$) that allows to synthesize person images in arbitrary poses, based on an image of that person and a novel pose. Our generation framework PG$^2$ utilizes the pose information explicitly and consists of two key stages: pose integration and image refinement. In the first stage the condition image and the target pose are fed into a U-Net-like network to generate an initial but coarse image of the person with the target pose. The second stage then refines the initial and blurry result by training a U-Net-like generator in an adversarial way. Extensive experimental results on both 128$\times$64 re-identification images and 256$\times$256 fashion photos show that our model generates high-quality person images with convincing details.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View (2019, background): “face frontal view generation [50], generating new human poses [51], photos to emojis [52], photograph editing [53]–[55], face aging [56], [57], photo blending [58], super resolution [59]–[61], photo inpainting [62]–[64], video prediction [65], and 3D object generation [66], [67]; among”
A Hybrid Model for Identity Obfuscation by Face Replacement (2018, methodology): “Convolutional Auto-encoder with skip connections between encoder and decoder(4), following [1][35][36].”
Lifelong GAN: Continual Learning for Conditional Image Generation (2019, background): “Models [38, 22] for rendering a person’s appearance onto a given pose have shown to be effective for person re-identification.”
Video-to-Video Synthesis (2018, background): “We note that the image counterpart was studied in recent works [4,17,47,48].”
Transferring GANs: generating images from limited data (2018, methodology): “GANs have further been widely applied in applications, including super-resolution [26], 3D object generation and reconstruction [40], human pose estimation [28], and age estimation [47].”
Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies (2021, background): “Based on the advances in image-to-image translation techniques [20], [38, 8, 39] train a network to map 2D skeleton images to target rendering results.”
Progressive Pose Attention Transfer for Person Image Generation (2019, methodology): “Following the protocol of [21, 34], 55 real and 55 generated images are randomly selected and shuffled, the first 10 of them are used for practice then the remaining 100 images are used as the evaluation set.”
Pose-Normalized Image Generation for Person Re-identification (2017, background): “This problem is tackled in a very recent work [22, 30].”
A Variational U-Net for Conditional Appearance and Shape Generation (2018, methodology): “Recently there has been great interest in generative models for image synthesis [7, 12, 18, 24, 49, 51, 32].”
Behavior-Driven Synthesis of Human Dynamics (2021, background): “Much work has been proposed to alter certain characteristics of humans depicted in static images like age, gender or body features [31, 33, 35, 52] or synthesizing individual persons in different, unseen poses [20, 19, 38, 13].”
Text as Neural Operator:Image Manipulation by Text Instruction (2020, background): “The input context can be images [18, 56, 24, 16, 37], audio sequences [25], human poses [34], or semantic segmentation masks [49, 41, 28].”
Synthesizing Images of Humans in Unseen Poses (2018, methodology): “Variations of a UNet architecture have been used in a variety of image synthesis works, including some focusing on human synthesis [11, 13].”
VITON: An Image-Based Virtual Try-on Network (2017, methodology): “Note that our representation is more detailed than previous work [32, 51].”
Deformable GANs for Pose-Based Human Image Generation (2017, methodology): “[12] propose to alleviate this problem using a two-stage generation approach.”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, background): “Field Subfield Method Super-resolution SRGAN [63], ESRGAN [64], Cycle-in-Cycle GANs [65], SRDGAN [66], TGAN [67] DR-GAN [68], TP-GAN [69], PG2 [70], PSGAN [71], Image synthesis and manipulation APDrawingGAN [72], IGAN [73], Image processing and computer vision introspective adversarial networks [74]”
Zero-Shot Visual Recognition Using Semantics-Preserving Adversarial Embedding Networks (2017, background): “We seek algorithms that can generate perceptually realistic images [16, 23, 5, 32, 33, 56].”
DRIT++: Diverse Image-to-Image Translation via Disentangled Representations (2019, background): “…recent efforts exploit conditional GAN in various contexts including conditioned on text (Reed et al. 2016), audio (Lee et al. 2019), low-resolution images (Ledig et al. 2017), human pose (Ma et al. 2017; AlBahar andHuang 2019), video frames (Vondrick et al. 2016), and image (Isola et al. 2017).”
Everybody Dance Now (2018, background): “Modern approaches have shown success in generating detailed single images of human subjects in new poses [3, 10, 11, 18, 22, 27, 28, 33, 38, 13, 46].”
Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics (2019, methodology): “Instead, we follow the face in-painting work [45] and use the Mask-SSIM score [32] as a referenced quantitative metric of visual quality of synthesized DeepFake video frames.”