U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation

By Minjae Kim, Hyeonwoo Kang, Kwanghee Lee. 2019

641 citations recorded by the source corpus.

We propose a novel method for unsupervised image-to-image translation, which incorporates a new attention module and a new learnable normalization function in an end-to-end manner. The attention module guides our model to focus on more important regions distinguishing between source and target domains based on the attention map obtained by the auxiliary classifier. Unlike previous attention-based method which cannot handle the geometric changes between domains, our model can translate both images requiring holistic changes and images requiring large shape changes. Moreover, our new AdaLIN (Adaptive Layer-Instance Normalization) function helps our attention-guided model to flexibly control the amount of change in shape and texture by learned parameters depending on datasets. Experimental results show the superiority of the proposed method compared to the existing state-of-the-art models with a fixed network architecture and hyper-parameters. Our code and datasets are available at this https URL or this https URL.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

AgileGAN (2021, methodology): “Preference Score ↑ FID ↓ Toonify[Pinkney and Adler 2020] 25.4 82.7 CycleGan[Zhu et al. 2017] 1.3 109.3 UNIT[Liu et al. 2017] 10.8 123.9 UGATIT[Kim et al. 2020] 4.6 104.7 Ours 57.9 64.7 instances from the imputed Gaussian distribution, leading to multiple output images.”
Enhancing Photorealism Enhancement (2021, background): “Notable improvements came from cycle-consistency [59], [62], custom attention modules [63], temporal regularization [64], modeling sensor noise [65], geometric constraints in the image plane [66], constraints derived from depth maps [67], and contrastive losses [60].”
A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters (2020, methodology): “We test two DNN models, UGATIT GAN [41] (one of the most popular models for image generation) and GPT-2 [57] (one of the most popular NLP models for text generation), both implemented in PyTorch.”
Feature Quantization Improves GAN Training (2020, methodology): “Follow (Kim et al., 2020), we use 120 (horse), 140 (zebra), 751 (photo), and 400 (vangogh) randomly selected images as test data, respectively.”
The Role of ImageNet Classes in Fréchet Inception Distance (2022, background): “…Razavi et al., 2019; Esser et al., 2021; Karras et al., 2019; 2020b;a; 2021), unsupervised domain translation (Zhu et al., 2017; Choi et al., 2020; Kim et al., 2020), image super resolution (Ledig et al., 2017; Bell-Kligler et al., 2019; Saharia et al., 2021), image editing (Park et al., 2019;…”
Alleviating Semantics Distortion in Unsupervised Low-Level Image-to-Image Translation via Structure Consistency Constraint (2022, methodology): “Effectiveness and Compatibility We couple our structure consistency constraint (SCC) with the vanilla GAN to show its effectiveness, and incorporate SCC with some popular methods such as CycleGAN [73], GcGAN [10], and U-GATIT [25] to show its compatibility.”
AgileGAN: Stylizing Portraits by Inversion-Consistent Transfer Learning (2021, methodology): “8, the results of our method can be compared to Toonify [Pinkney and Adler 2020] and other recent unpaired image translation techniques, including CycleGan [Zhu et al. 2017], UNIT [Liu et al. 2017] and UGATIT [Kim et al. 2020].”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “…layer of the LSTM. Similar ideas are also used in adaptive instance normalization (AdaIN) [69] and adaptive layer-instance normalization (AdaLIN) [102] for unsupervised image-to-image translation, where the subnetworks are MLPs and the inputs of the subnetworks are the embedding features…”
Scaling Distributed Machine Learning with In-Network Aggregation (2019, neutral): “…PyT NCF [39] Recommendation ML-20m [36] PyT BERT [24] Question Answering SQuAD [81] TF VGG19 [91] Image Classiﬁcation ImageNet-1K [85] PyT UGATIT [51] Image-to-Image Translation Selﬁe2Anime [51] TF ResNet50 [38] Image Classiﬁcation ImageNet-1K [85] TF SSD [63] Object Detection COCO 2017 [59] PyT…”
Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes (2021, background): “cess is confined to the pixel domain, almost as if style transfer is a special case of image-to-image translation [21, 58, 51, 36, 19, 37, 6, 7, 25].”