Disentangled Person Image Generation

By Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, Mario Fritz. 2017

458 citations recorded by the source corpus.

Generating novel, yet realistic, images of persons is a challenging task due to the complex interplay between the different image factors, such as the foreground, background and pose information. In this work, we aim at generating such images based on a novel, two-stage reconstruction pipeline that learns a disentangled representation of the aforementioned image factors and generates novel person images at the same time. First, a multi-branched reconstruction network is proposed to disentangle and encode the three factors into embedding features, which are then combined to re-compose the input image itself. Second, three corresponding mapping functions are learned in an adversarial manner in order to map Gaussian noise to the learned embedding feature space, for each factor, respectively. Using the proposed framework, we can manipulate the foreground, background and pose of the input image, and also sample new embedding features to generate such targeted manipulations, that provide more control over the generation process. Experiments on the Market-1501 and Deepfashion datasets show that our model does not only generate realistic person images with new foregrounds, backgrounds and poses, but also manipulates the generated factors and interpolates the in-between states. Another set of experiments on Market-1501 shows that our model can also be beneficial for the person re-identification task1.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Hybrid Model for Identity Obfuscation by Face Replacement (2018, positive): “enerator (G) and Discriminator (D). The head generator Gis a \U-Net"-based architecture [34], i.e. Convolutional Auto-encoder with skip connections between encoder and decoder4, following [1][35][36]. It generates a natural head image given both the surrounding context and the rendered face. Th”
Video-to-Video Synthesis (2018, background): “We note that the image counterpart was studied in recent works [4,17,47,48].”
A Disentangling Invertible Interpretation Network for Explaining Latent Representations (2020, methodology): “In contrast to existing disentangled image synthesis [34, 8, 32, 24, 7], our invertible approach can be applied on top of existing autoencoder representations, which therefore do not have to be altered or retrained to handle different semantic concepts.”
Person Transfer GAN to Bridge Domain Gap for Person Re-identification (2017, background): “[7], many variants of GAN [24, 25, 30, 36, 17, 34, 16, 5, 21, 35, 14, 42, 23] have been proposed to tackle different tasks, e.”
Zero-Shot Visual Recognition Using Semantics-Preserving Adversarial Embedding Networks (2017, background): “We seek algorithms that can generate perceptually realistic images [16, 23, 5, 32, 33, 56].”
The Creation and Detection of Deepfakes (2020, methodology): “e authors of [46] then streamlined this process by using a single ED GAN network to disentangle the foreground appearance (body), background appearance, and pose.”
Everybody Dance Now (2018, background): “Modern approaches have shown success in generating detailed single images of human subjects in new poses [3, 10, 11, 18, 22, 27, 28, 33, 38, 13, 46].”
Natural and Effective Obfuscation by Head Inpainting (2017, background): “Prior work on structure-guided image generation has shown that such a guidance is indeed very helpful for generating images with complex inner structures (e.g. persons) [13, 5, 28, 30, 6, 33, 14, 2].”
Image-to-Image Translation: Methods and Applications (2021, unknown): “Person image synthesis, including virtual try-on [180], [181], [182], [183], [184], [185] and skeleton-to-person translation [117], [118], learns”
Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos (2020, methodology): “[15], [16] incorporates foreground-background separation with 2D keypoints annotations and applying a set of morphological operations to those keypoints such that it is able to approximately cover the whole human body.”
Learning to Reduce Dual-Level Discrepancy for Infrared-Visible Person Re-Identification (2019, background): “[13] manipulated the foreground, background and pose information, and generated images based on manipulated information.”
Behavior-Driven Synthesis of Human Dynamics (2021, background): “[40] Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool,”
Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video (2020, background): “Several approaches address related problems to ours, such as generating images of humans in new poses [2, 41, 49, 64] or body reenactment from monocular videos [8].”
Hi-CMD: Hierarchical Cross-Modality Disentanglement for Visible-Infrared Person Re-Identification (2019, background): “Among them, some previous works in the single-modality person re-identification task have been conducted with a purpose of disentangling foreground, background, and pose factors [20], or extracting illumination-invariant features [40].”
Adaptive Appearance Rendering (2021, background): “Generative models of video sequences [28, 29] and fashion images [17, 31] are a core part of visual understanding and have received renewed attention from the vision community.”
Cross-Modality Paired-Images Generation for RGB-Infrared Person Re-Identification (2020, methodology): “Ma et al. (Ma et al. 2018) use a cGAN to generate pedestrian images with different poses to learn features free of inﬂuences of pose variation.”
Image Generation From Layout (2018, background): “In [24], three factors (foreground, background and pose) are disentangled explicitly when generating person image.”
Network-to-Network Translation with Conditional Invertible Neural Networks (2020, background): “Examples for conditioning images include grayscale images [84], low resolution images [38], edge images [33], segmentation maps [51, 7] or heatmaps of keypoints [23, 43].”
VERI-Wild: A Large Dataset and a New Method for Vehicle Re-Identification in the Wild (2019, methodology): “Some other methods focus on image transferring between different datasets [22][3] or generating different human poses [13], but they are not suitable for vehicles.”