Cross-Domain Correspondence Learning for Exemplar-Based Image Translation

By Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, Fang Wen. 2020

266 citations recorded by the source corpus.

We present a general framework for exemplar-based image translation, which synthesizes a photo-realistic image from the input in a distinct domain (e.g., semantic segmentation mask, or edge map, or pose keypoints), given an exemplar image. The output has the style (e.g., color, texture) in consistency with the semantically corresponding objects in the exemplar. We propose to jointly learn the cross-domain correspondence and the image translation, where both tasks facilitate each other and thus can be learned with weak supervision. The images from distinct domains are first aligned to an intermediate domain where dense correspondence is established. Then, the network synthesizes images based on the appearance of semantically corresponding patches in the exemplar. We demonstrate the effectiveness of our approach in several image translation tasks. Our method is superior to state-of-the-art methods in terms of image quality significantly, with the image style faithful to the exemplar with semantic consistency. Moreover, we show the utility of our method for several applications.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Taming Transformers for High-Resolution Image Synthesis (2020, background): “This inductive bias towards local interactions thus leads to efficient computations, but the wide range of specialized layers which are introduced into CNNs to handle different synthesis tasks [46, 70, 59, 74, 73] suggest that this bias is often too restrictive.”
Image-to-Image Translation: Methods and Applications (2021, unknown): “Sketch-to-image [1], [87], [114], [117], [118] [15], [120] [102], [10], [2], [133], [147] [79], [150], [90], [76], [154] \ \ [169] [175] \ \ \”
Deep Sketch-Guided Cartoon Video Inbetweening (2021, methodology): “Moreover, recent image to image translation techniques can also be used to translate sketches to cartoon images [7], [8], [19].”
Network-to-Network Translation with Conditional Invertible Neural Networks (2020, result): “This stands in contrast to current inﬂuential domain transfer approaches [41, 51, 83] that require learning or ﬁnetuning existing domain representations to facilitate transfer between them.”
Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation (2021, background): “As suggested by the theoretical analysis [3], domain alignment methods focus on reducing the distribution mismatch by optimizing some divergence [30, 35] or adopting adversarial training [20, 40] at either the image level [1, 12, 19, 26, 49, 63, 68], the intermediate feature level [6, 27, 48, 58] or”
Pretraining is All You Need for Image-to-Image Translation (2022, background): “The most popular choice is to use conditional generative adversarial networks [23, 63, 50, 35, 8, 59, 62] (cGAN) that rely on a discriminator to examine the gap with real images.”
Paint by Example: Exemplar-based Image Editing with Diffusion Models (2022, background): “Another popular direction of works [5, 18, 36, 65, 73, 75] utilize semantic mask to control the editing.”
Adding Conditional Control to Text-to-Image Diffusion Models (2023, methodology): “Pix2Pix [20] presented the concept of image-to-image translation, and early methods are dominated by conditional generative neural networks [20, 69, 60, 39, 8, 63, 68].”