Contrastive Learning for Unpaired Image-to-Image Translation

By Taesung Park, Alexei A. Efros, Richard Zhang, Jun-Yan Zhu. 2020

1,660 citations recorded by the source corpus.

In image-to-image translation, each patch in the output should reflect the content of the corresponding patch in the input, independent of domain. We propose a straightforward method for doing so -- maximizing mutual information between the two, using a framework based on contrastive learning. The method encourages two elements (corresponding patches) to map to a similar point in a learned feature space, relative to other elements (other patches) in the dataset, referred to as negatives. We explore several critical design choices for making contrastive learning effective in the image synthesis setting. Notably, we use a multilayer, patch-based approach, rather than operate on entire images. Furthermore, we draw negatives from within the input image itself, rather than from the rest of the dataset. We demonstrate that our framework enables one-sided translation in the unpaired image-to-image translation setting, while improving quality and reducing training time. In addition, our method can even be extended to the training setting where each "domain" is only a single image.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

Image-to-Image Translation: Methods and Applications (2021, background): “style transfer artistic:[113], [1] artistic:[15] artistic:[2], [36], [136], [137], [145]; photo-realistic: [102], [139], [2], [75], [136], [137], artistic:[152]; photo-realistic: [90], [76], [77], [77], [143], [152], [154] \ artistic:[155], [81], [166]; photo-realistic: [155], [156], [81] artistic:[”
Dual Contrastive Loss and Attention for GANs (2021, methodology): “GAN techniques have been popularized into extensive computer vision applications, including but not limited to image translation [35, 107, 108, 54, 33, 82, 64, 20, 63], postDual Contrastive Loss Pr im ar y Im ag e”
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization (2021, background): “1 Contrastive learning (CL) has obtained impressive results on many visual processing tasks, such as image classiﬁcation (Khosla et al., 2020; Chen et al., 2020) and synthesis (Park et al., 2020; Zhang et al., 2021b).”
Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis (2021, background): “The self-supervised contrastive coding presents promising performance on visual representation learning when transferred to downstream tasks [10, 36].”
Data-Efficient Instance Generation from Instance Discrimination (2021, background): “Many attempts have been made to improve generative models by drawing lessons from contrastive learning, like the consistency regularization for GANs [47, 50], the patch-level contrastive learning for image-to-image translation [33], and the latent-augmented contrastive loss for conditional image syn”
StyleAlign: Analysis and Applications of Aligned StyleGAN Models (2021, methodology): “In Figure 5 we compare our I2I results to two state-of-the-art methods, CUT (Park et al., 2020) and F-LSeSim (Zheng et al., 2021).”
StyTr2: Image Style Transfer with Transformers (2021, background): “However, strict reversibility may not be suitable for generation tasks [40].”
Contrastive Learning for Compact Single Image Dehazing (2021, methodology): “Moreover, different from [33], we proposed a new sampling method and a novel pixel-wise contrastive loss (a.”
Divide and Contrast: Self-supervised Learning from Uncurated Data (2021, background): “Finally, contrastive learning has also been applied to supervised image classification [48], image translation [70], knowledge distillation [89, 77], and adversarial learning [49].”
Scaling up GANs for Text-to-Image Synthesis (2023, neutral): “As the sampling quality and diversity of GANs improve [31–34, 36, 51, 60], GANs have been deployed to various computer vision and graphics applications, such as text-to-image synthesis [55], image-toimage translation [23, 27, 37, 47, 48, 82], and image editing [1,6,49,81].”
ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models (2021, background): “Here, diverse samples are generated, however, some existing image translation models fail [17, 30] to produce stochastic samples.”
Artificial Fingerprinting for Generative Models: Rooting Deepfake Attribution in Training Data (2020, background): “Successes have also spread to many other vision tasks (e.g. [38, 30, 24, 63, 64, 37, 52]).”
Contrastive Learning for Diverse Disentangled Foreground Generation (2022, background): “Recently [31] proposes to use contrastive learning in image translation task.”
Language-Driven Image Style Transfer (2021, methodology): “For computer vision, they involve data augmentation [37, 38, 31] or patch-wise comparison [33, 40] to provide contrastive loss.”
Unsupervised Degradation Representation Learning for Blind Super-Resolution (2021, background): “As emphasized in existing contrastive learning methods [5, 17, 31], a large dictionary covering a rich set of negative samples is critical for good representation learning.”