A Variational U-Net for Conditional Appearance and Shape Generation

By Patrick Esser, Ekaterina Sutter, Björn Ommer. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

448 citations recorded by the source corpus.

Deep generative models have demonstrated great performance in image synthesis. However, results deteriorate in case of spatial deformations, since they generate images of objects directly, rather than modeling the intricate interplay of their inherent shape and appearance. We present a conditional U-Net [30] for shape-guided image generation, conditioned on the output of a variational autoencoder for appearance. The approach is trained end-to-end on images, without requiring samples of the same object with varying pose or appearance. Experiments show that the model enables conditional image generation and transfer. Therefore, either shape or appearance can be retained from a query image, while freely altering the other. Moreover, appearance can be sampled due to its stochastic latent representation, while preserving shape. In quantitative and qualitative experiments on COCO [20], DeepFashion [21, 23], shoes [43], Market-1501 [47] and handbags [49] the approach demonstrates significant improvements over the state-of-the-art.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/5fdb3533152f9862e3e4c2282cd5f1400af18956

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

First Order Motion Model for Image Animation (2020, result): “Considering an externally trained image representation, we report the average euclidean distance between the ground truth and generated frame representation, similarly to Esser et al. [11].”
A Style-Aware Content Loss for Real-time HD Style Transfer (2018, background): “Other works like [48,5,34,9,1] learn how to discriminate different techniques, styles and contents in the latent space.”
MakeItTalk: Speaker-Aware Talking Head Animation (2020, background): “Its architecture follows the generators proposed in Esser et al. [2018] and Han et al. [2019].”
A Probabilistic U-Net for Segmentation of Ambiguous Images (2018, background): “Very recent work on generating appearances given a shape encoding [22] also combines a U-Net with a VAE, and was developed concurrently to ours.”
Video-to-Video Synthesis (2018, background): “We note that the image counterpart was studied in recent works [4,17,47,48].”
ClothFlow: A Flow-Based Model for Clothed Person Generation (2019, methodology): “[8] disentangled the pose and appearance generation with a variational U-NET.”
A Disentangling Invertible Interpretation Network for Explaining Latent Representations (2020, methodology): “In contrast to existing disentangled image synthesis [34, 8, 32, 24, 7], our invertible approach can be applied on top of existing autoencoder representations, which therefore do not have to be altered or retrained to handle different semantic concepts.”
Variational Convolutional Neural Network Pruning (2019, methodology): “We set the variance of prior and approximated posterior to be identical for simplifying the KL loss, which beneﬁts for training [8].”
Animating Arbitrary Objects via Deep Motion Transfer (2018, methodology): “We compute the feature-based metric employed in [12] that consists in computing the Average Euclidean Distance (AED) between a feature representation of the ground truth and the generated video frames.”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, methodology): “Scandalea et al. [109] proposed a framework based on U-Net structure for deep AL to segment biomedical images, and used uncertainty measure obtained by MC dropout, to suggest the sample to be annotated.”
Everybody Dance Now (2018, background): “Modern approaches have shown success in generating detailed single images of human subjects in new poses [3, 10, 11, 18, 22, 27, 28, 33, 38, 13, 46].”
MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generation (2019, background): “Others require strong supervision in the form of keypoint/pose/mask annotations [31, 1, 27, 9], which limits their scalability, and still fall short of disentangling all of the four factors outlined above.”
Mutual consistency learning for semi-supervised medical image segmentation (2021, methodology): “Here, each sub-model f θ is a standard encoder-decoder architecture like V-Net or U-Net (Milletari et al., 2016; Ronneberger et al., 2015).”
Disentangled representation learning in cardiac image analysis (2019, methodology): “Furthermore, Esser et al. [2018] expressed content as a shape estimation (using an edge extractor and a pose estimator) and combined it with style obtained from a VAE.”
A Hybrid Video Anomaly Detection Framework via Memory-Augmented Flow Reconstruction and Flow-Guided Frame Prediction (2021, methodology): “Inspired by the Variational UNet proposed in [6], we add skip connections between Fφ and Dψ to help generating xt+1.”
Network-to-Network Translation with Conditional Invertible Neural Networks (2020, background): “Examples for conditioning images include grayscale images [84], low resolution images [38], edge images [33], segmentation maps [51, 7] or heatmaps of keypoints [23, 43].”
Progressive Pose Attention Transfer for Person Image Generation (2019, methodology): “We further compare our method to several existing person image generators [5, 34] under the same setting for re-ID data augmentation.”
Single-Shot Freestyle Dance Reenactment (2020, background): “In still images, the problem of pose transfer is well studied [28, 37, 1, 45, 9, 57, 11, 10, 39, 30, 26], out of which [9, 39, 30] use a human parser, as we do.”
Behavior-Driven Synthesis of Human Dynamics (2021, background): “Much work has been proposed to alter certain characteristics of humans depicted in static images like age, gender or body features [31, 33, 35, 52] or synthesizing individual persons in different, unseen poses [20, 19, 38, 13].”
Image-to-Image Translation: Methods and Applications (2021, unknown): “Person image synthesis, including virtual try-on [180], [181], [182], [183], [184], [185] and skeleton-to-person translation [117], [118], learns”