Deformable GANs for Pose-Based Human Image Generation

By Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuilière, Nicu Sebe. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017

496 citations recorded by the source corpus.

In this paper we address the problem of generating person images conditioned on a given pose. Specifically, given an image of a person and a target pose, we synthesize a new image of that person in the novel pose. In order to deal with pixel-to-pixel misalignments caused by the pose differences, we introduce deformable skip connections in the generator of our Generative Adversarial Network. Moreover, a nearest-neighbour loss is proposed instead of the common L1 and L2 losses in order to match the details of the generated image with the target image. We test our approach using photos of persons in different poses and we compare our method with previous work in this area showing state-of-the-art results in two benchmarks. Our method can be applied to the wider field of deformable object generation, provided that the pose of the articulated object can be extracted using a keypoint detector.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/74b9632e8c7bc7c96af5561a017b40b9613f196d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Image-to-Image Translation: Methods and Applications (2021, unknown): “Person image synthesis, including virtual try-on [180], [181], [182], [183], [184], [185] and skeleton-to-person translation [117], [118], learns”
Deep 3D human pose estimation: A review (2021, unknown): “…can also be used to assist other computer vision tasks such as pose transfer (Li et al., 2019a), action recognition (Luvizon et al., 2018), human parsing (Xia et al., 2016), person image generation (Siarohin et al., 2018), animation (Weng et al., 2019), pose search (Ferrari et al., 2009).”
Thin-Plate Spline Motion Model for Image Animation (2022, methodology): “However, these approaches, which rely on labeled data, only work for specific objects, such as faces [9, 11, 30, 35] and human bodies [6, 18, 23, 27, 37].”
Whitening and Coloring transform for GANs (2018, background): “[8] extend this framework to deal with structured conditional information (the human body pose).”
Everybody Dance Now (2018, background): “[33] have introduced novel architectures and losses for this purpose.”
Progressive Pose Attention Transfer for Person Image Generation (2019, methodology): “[34] introduced deformable skip connections that require exten-”
The Creation and Detection of Deepfakes (2020, background): “Although the work of [146] helps align the general images, artifacts can still occur when xs and xt have very dierent poses.”
Animating Arbitrary Objects via Deep Motion Transfer (2018, methodology): “Motivated by [31], where it was shown that a standard U-net cannot handle large pixel-to-pixel misalignment between the input and the output images, we propose using a deformation module to align the features of the encoder with the output images.”
ClothFlow: A Flow-Based Model for Clothed Person Generation (2019, methodology): “DSC [36] VUNET [8] DPT [30] CBI [12] Soft-Gated [4]”
First Order Motion Model for Image Animation (2020, methodology): “In order to handle this misalignment, we use a feature warping strategy similar to [30, 29, 15].”
Joint Discriminative and Generative Learning for Person Re-Identification (2019, methodology): “achieve better poseconditioned image generation by using a nearest neighbor loss to replace the traditional 1 or 2 loss [33].”
Toward Characteristic-Preserving Image-based Virtual Try-On Network (2018, methodology): “The generated fashion items in [37, 23, 31], kept consistent with that of the conditional images.”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, background): “[326] proposed deformable gans for pose-based human image generation.”
FD-GAN: Pose-guided Feature Distilling GAN for Robust Person Re-identification (2018, methodology): “Comparison of person image generation [18, 19].”
Single-Shot Freestyle Dance Reenactment (2020, background): “In still images, the problem of pose transfer is well studied [28, 37, 1, 45, 9, 57, 11, 10, 39, 30, 26], out of which [9, 39, 30] use a human parser, as we do.”
Motion Representations for Articulated Animation (2021, background): “However in contrast to FOMM [27], but similar to Monkey-Net [28], here deformable skip connections [30] are used between the encoder and decoder.”
Whitening and Coloring Batch Transform for GANs (2018, background): “Siarohin et al. (2018a) extend this framework to deal with structured conditional information (the human body pose).”