Generative Semantic Manipulation with Contrasting GAN

By Xiaodan Liang, H. Zhang, Eric P. Xing. 2017

53 citations recorded by the source corpus.

Generative Adversarial Networks (GANs) have recently achieved significant improvement on paired/unpaired image-to-image translation, such as photo$\rightarrow$ sketch and artist painting style transfer. However, existing models can only be capable of transferring the low-level information (e.g. color or texture changes), but fail to edit high-level semantic meanings (e.g., geometric structure or content) of objects. On the other hand, while some researches can synthesize compelling real-world images given a class label or caption, they cannot condition on arbitrary shapes or structures, which largely limits their application scenarios and interpretive capability of model results. In this work, we focus on a more challenging semantic manipulation task, which aims to modify the semantic meaning of an object while preserving its own characteristics (e.g. viewpoints and shapes), such as cow$\rightarrow$sheep, motor$\rightarrow$ bicycle, cat$\rightarrow$dog. To tackle such large semantic changes, we introduce a contrasting GAN (contrast-GAN) with a novel adversarial contrasting objective. Instead of directly making the synthesized samples close to target data as previous GANs did, our adversarial contrasting objective optimizes over the distance comparisons between samples, that is, enforcing the manipulated data be semantically closer to the real data with target category than the input data. Equipped with the new contrasting objective, a novel mask-conditional contrast-GAN architecture is proposed to enable disentangle image background with object semantic changes. Experiments on several semantic manipulation tasks on ImageNet and MSCOCO dataset show considerable performance gain by our contrast-GAN over other conditional GANs. Quantitative results further demonstrate the superiority of our model on generating manipulated results with high visual fidelity and reasonable object semantics.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

The Perception-Distortion Tradeoff (2017, background): “-fake user studies, which examine the ability of human observers to tell whether x̂ is real or the output of an algorithm [48, 14, 6, 13, 49, 22, 10] (similarly to the idea underlying generative adversarial nets [9]).”
U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation (2019, methodology): “Unlike previous attention-based methods [29, 24] which cannot handle the geometric changes between domains, our model can translate both images requiring holistic changes and images requiring large shape changes.”
InstaGAN: Instance-aware Image-to-Image Translation (2018, background): “Mask contrast-GAN (Liang et al., 2017) and Attention-GAN (Mejjati et al., 2018) use segmentation masks or predicted attentions, but only to attach the background to the (translated) cropped instances.”
Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language (2018, background): “Automatic image manipulation can remedy this difﬁculty by automatically changing various aspects of images from low-level color/texture [1, 2] to high-level semantics [3] without tedious human operation.”
Synthetic Data for Deep Learning (2019, background): “nd image-to-image translation [384]. In general, these works belong to an interesting eld of generative semantic manipulation with GANs. Important works in this direction include Mask-Contrasting GAN [358] that can modify an object to a dierent suitable category inside its segmentation mask (e.g., r”
Toward Characteristic-Preserving Image-based Virtual Try-On Network (2018, background): “image-to-image translation [12,38,5,34,20,6]) make it seem to be a natural approach of facilitating this problem.”
Improving Shape Deformation in Unsupervised Image-to-Image Translation (2018, methodology): “Later approaches [46,24] used a patch-based DCGAN [35] discriminator, initially developed for style transfer and texture synthesis [23].”