Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks
By Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros. 2017
5,640 citations recorded by the source corpus.
Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs. However, for many tasks, paired training data will not be available. We present an approach for learning to translate an image from a source domain X to a target domain Y in the absence of paired examples. Our goal is to learn a mapping G : X → Y such that the distribution of images from G(X) is indistinguishable from the distribution Y using an adversarial loss. Because this mapping is highly under-constrained, we couple it with an inverse mapping F : Y → X and introduce a cycle consistency loss to push F(G(X)) ≈ X (and vice versa). Qualitative results are presented on several tasks where paired training data does not exist, including collection style transfer, object transfiguration, season transfer, photo enhancement, etc. Quantitative comparisons against several prior methods demonstrate the superiority of our approach.
Subfields: Computer Vision and Pattern Recognition, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Yi Yang
- Jitendra Malik
- Andrew Zisserman
In Generative Models
- Jun-Yan Zhu
- Richard Zhang
- Eli Shechtman
- Daniel Cohen-Or
- Stefano Ermon
- Jiaming Song
- Arash Vahdat
- Karsten Kreis
Citations in the corpus
- SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, background): “Wang and Lee (2018) trained a Cycle-GAN model (Zhu et al., 2017) to learn a documentto-summary mapping given large datasets of unpaired documents and summaries.”
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “CycleGAN [71] trained a mapping function from one domain to the other.”
- Projected GANs Converge Faster (2021, background): “They show that their approach improves CycleGAN [92] on image translation tasks.”
- VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, background): “, 2016a) generation, distribution alignment across domains (Zhu et al., 2017; Liu et al., 2017; Tzeng et al., 2017) and semi-supervised learning (Kingma et al.”
- Imperfect ImaGANation: Implications of GANs Exacerbating Biases on Facial Data Augmentation and Snapchat Selfie Lenses (2020, background): “Accepted for publication in the ICLR 2021 Workshop on Synthetic Data Generation – Quality, Privacy, Bias To study this, we train a CycleGAN (Zhu et al., 2017) to stylize faces of non-engineering professors to look like engineering faculty.”
- Improved Precision and Recall Metric for Assessing Generative Models (2019, background): “We believe that the separate quantification of precision can also be useful in the context of image-to-image translation [33], where the quality of individual images is of great interest.”
- Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, background): “Another widely-used form is image-to-image translation [18,20,23, 27, 49, 50], where both input and output are images.”
- EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning (2019, methodology): “CycleGAN [57] extends this framework and finds a reverse mapping back to the original data distribution.”
- GAN Inversion: A Survey (2021, methodology): “In recent years, GANs have been applied to computer vision tasks ranging from image translation [2], [3], [4], image manipulation [5], [6], [7], to image restoration [8], [9], [10].”
- High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, methodology): “ as transforming Google maps to satellite views and generating cats from user sketches. Various methods have also been proposed to learn an image-to-image translation in the absence of training pairs [2,34,35,47,50,52,56,65]. Recently, Chen and Koltun [5] suggest that it might be hard for conditiona”
- StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation (2017, methodology): “We leverage PatchGANs [7, 15, 33] for the discriminator network, which classifies whether local image patches are real or fake.”
- Unsupervised Image-to-Image Translation Networks (2017, methodology): “arly, x 1= F 2!(x 2) = G (E(x 2)). The UNIT problem then becomes a problem of learning F1! and F2!1 . We note that a necessary condition for F1!2 and F2!1 to exist is the cycle-consistency constraint [29, 10]: x 1= F 2!(F (x 1)) and x2 = F!2(F (x2)). We can reconstruct the input image from translati”
- InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, neutral): “Generative models for image editing Image editing models traditionally targeted a single editing task such as style transfer [15, 16] or translation between image domains [22, 24, 36, 42, 71].”
- Nucleus segmentation across imaging experiments: The 2018 Data Science Bowl (2019, neutral): “The 2018 Data Science Bowl presented the challenge of automatically finding nuclei in a large variety of unseen microscopy images, with no configuration step.”
- Rewriting a Deep Generative Model (2020, background): “At a practical level, deep generative models are increasingly useful for image and video synthesis [57,86,35,12].”
- Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization (2018, methodology): “To address this issue, inspired by recent work on leveraging “cycle consistency” for image generation [13, 25], we implement a dual objective that treats source and target equally, by complementing the objective in Eqn.”