Pretraining is All You Need for Image-to-Image Translation

By Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, Fang Wen. 2022

216 citations recorded by the source corpus.

We propose to use pretraining to boost general image-to-image translation. Prior image-to-image translation methods usually need dedicated architectural design and train individual translation models from scratch, struggling for high-quality generation of complex scenes, especially when paired training data are not abundant. In this paper, we regard each image-to-image translation problem as a downstream task and introduce a simple and generic framework that adapts a pretrained diffusion model to accommodate various kinds of image-to-image translation. We also propose adversarial training to enhance the texture synthesis in the diffusion model training, in conjunction with normalized guidance sampling to improve the generation quality. We present extensive empirical comparison across various tasks on challenging benchmarks such as ADE20K, COCO-Stuff, and DIODE, showing the proposed pretraining-based image-to-image translation (PITI) is capable of synthesizing images of unprecedented realism and faithfulness.

Subfields: Cancer Research, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cancer Research

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “We minimize the following latent diffusion objective: Wang et al. [66] show that fine-tuning a large image diffusion models outperforms training a model from scratch for image translation tasks, especially when paired training data is limited.”
RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion (2022, background): “The most remarkable success happens in text-to-image synthesis [19, 39, 47, 50, 52], which serves as a foundation model and enables various appealing applications [21, 51, 66] previously unattainable.”
Paint by Example: Exemplar-based Image Editing with Diffusion Models (2022, background): “Another popular direction of works [5,18,36,65,73,75] utilize semantic mask to control the editing.”
Ablating Concepts in Text-to-Image Diffusion Models (2023, background): “Fine-tuning aims to adapt the weights of a pretrained generative model to new domains [73, 46, 72, 41, 79, 34, 47, 80, 30, 35, 24, 44], downstream tasks [71, 54, 78], and test images [6, 53, 48, 31, 25, 49].”
Composer: Creative and Controllable Image Synthesis with Composable Conditions (2023, methodology): “…al., 2022), scene graphs (Yang et al., 2022), sketches (Voynov et al., 2022), depthmaps (stability.ai, 2022), and inpainting masks (Xie et al., 2022; Wang et al., 2022a), or by ﬁnetuning the pretrained models on a few subject-speciﬁc data (Gal et al., 2022; Mokady et al., 2022; Ruiz et al., 2022).”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, methodology): “Most existing methods typically achieve controllable generation mainly by introducing new conditions, such as segmentation maps [44, 59], inpainting masks [66] or sketches [34, 72], in addition to texts.”
Zero-shot Image-to-Image Translation (2023, background): “2022], PITI [Wang et al. 2022]) learn conditional diffusionmodels tailored for image-to-image translation tasks.”
DiffEdit: Diffusion-based semantic image editing with mask guidance (2022, unknown): “Wang et al. (2022b) use DDIM encoding of the input image, and then decode on edited sketches or semantic segmentation maps.”
MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation (2023, background): “…(i) training a model from scratch or finetuning a given diffusion model for the task at hand (e.g., inpainting, layout-to-image training, etc. (Wang et al., 2022a; Ramesh et al., 2022; Rombach et al., 2022; Nichol et al., 2021; Avrahami et al., 2022b; Brooks et al., 2022; Wang et al., 2022b)).”
Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation (2022, methodology): “Several concurrent methods have taken first steps in controlling different properties of the generated content [14,21,39,48,50].”
Pix2Video: Video Editing using Image Diffusion (2023, methodology): “These methods work by either finetuning [54] an existing model, introducing adapter layers [32] or other trainable modules [53, 62], or utilizing an ensemble of denoising networks [2].”
Diffusion Models in Vision: A Survey (2022, methodology): “[36] train the GLIDE model [14] to obtain a rich semantic latent space.”
SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model (2022, unknown): “, unconditional image generation [7, 8, 25, 26], text-toimage generation [18–21], video generation [6], image inpainting [1,2,14,16], image translation [15,27,30], and image editing [4, 5, 10].”
Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior (2023, background): “Unlike the scarcity of 3D models, images are much more readily available, and recent advancements in diffusion models have sparked a revolution in 2D image generation [34, 39, 38, 52, 4].”
U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation (2024, background): “…prowess, from high-level detail to diversity in generated samples, has propelled breakthrough progress in various computer vision applications, such as image synthesis [26, 70, 74], image editing [3, 11, 46, 59], image-to-image translation [11, 45, 73, 85], and video generation [6, 25, 27, 41].”
Text-to-image Diffusion Models in Generative AI: A Survey (2023, background): “Another work [147] proposes to regard the image-to-image translation problem as a downstream task and successfully synthesizes images of unprecedented realism and faithfulness based on a pretrained diffusion model, termed as pretraining-based image-to-image translation (PITI) [147].”
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, background): “Applications of text-to-image diffusion models Apart from serving as a backbone to be fine-tuned for general image-to-image translation tasks [80], text-to-image diffusion models have also demonstrated impressive capabilities in other downstream applications.”
T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models (2023, background): “, some GAN-base [25, 35] and diffusion-based [43] methods.”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “However, there still exist some scenarios remaining underexplored [211], such as text-to-audiovisual speech synthesis [37] and visual question answering [5].”
Adding Conditional Control to Text-to-Image Diffusion Models (2023, methodology): “PITI [59] is a diffusion-based image-to-image translation method that utilizes large-scale pretraining as a way to improve the quality of generated results.”