InstaGAN: Instance-aware Image-to-Image Translation

By Sangwoo Mo, Minsu Cho, Jinwoo Shin. 2018

168 citations recorded by the source corpus.

Unsupervised image-to-image translation has gained considerable attention due to the recent impressive progress based on generative adversarial networks (GANs). However, previous methods often fail in challenging cases, in particular, when an image has multiple target instances and a translation task involves significant changes in shape, e.g., translating pants to skirts in fashion images. To tackle the issues, we propose a novel method, coined instance-aware GAN (InstaGAN), that incorporates the instance information (e.g., object segmentation masks) and improves multi-instance transfiguration. The proposed method translates both an image and the corresponding set of instance attributes while maintaining the permutation invariance property of the instances. To this end, we introduce a context preserving loss that encourages the network to learn the identity function outside of target instances. We also propose a sequential mini-batch inference/training technique that handles multiple instances with a limited GPU memory and enhances the network to generalize better for multiple instances. Our comparative evaluation demonstrates the effectiveness of the proposed method on different image datasets, in particular, in the aforementioned challenging cases. Code and results are available in this https URL

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “Instance segmentation has also been phrased as an image-to-image translation problem using conditional generative adversarial networks (Mo et al., 2019).”
Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs (2020, background): “, image generation [5, 20, 21], image-to-image translation [28, 33, 8], and video-to-video synthesis [43, 3, 6].”
Cali-Sketch: Stroke Calibration and Completion for High-Quality Face Image Generation from Poorly-Drawn Sketches (2019, background): “InstaGAN [43] incorporates instance attribute information for multi-instance transﬁguration.”
DiffEdit: Diffusion-based semantic image editing with mask guidance (2022, unknown): “…involve training an end-to-end architecture with a proxy objective before being adapted to editing at inference time, based on GANs (Li et al., 2020b;a; Ma et al., 2018; Alami Mejjati et al., 2018; Mo et al., 2018) or transformers (Wang et al., 2022a; Brown et al., 2022; Issenhuth et al., 2021).”
ManiGAN: Text-Guided Image Manipulation (2019, background): “Our work is related to conditional image synthesis [1, 19, 22, 24, 32, 33, 41].”
Image-to-Image Translation: Methods and Applications (2021, methodology): “[142] propose InstaGAN, which is the first work to solve multi-instance transfiguration tasks in UI2I.”