Adding Conditional Control to Text-to-Image Diffusion Models

By Lvmin Zhang, Anyi Rao, Maneesh Agrawala. 2023

7,354 citations recorded by the source corpus.

We present ControlNet, a neural network architecture to add spatial conditioning controls to large, pretrained text-to-image diffusion models. ControlNet locks the production-ready large diffusion models, and reuses their deep and robust encoding layers pretrained with billions of images as a strong backbone to learn a diverse set of conditional controls. The neural architecture is connected with "zero convolutions" (zero-initialized convolution layers) that progressively grow the parameters from zero and ensure that no harmful noise could affect the finetuning. We test various conditioning controls, e.g., edges, depth, segmentation, human pose, etc., with Stable Diffusion, using single or multiple conditions, with or without prompts. We show that the training of ControlNets is robust with small (<50k) and large (>1m) datasets. Extensive results show that ControlNet may facilitate wider applications to control image diffusion models.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Statistical and Nonlinear Physics, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Ming-Hsuan Yang
  3. Dacheng Tao
  4. Xiaogang Wang
  5. Trevor Darrell
  6. Chen Change Loy
  7. Jitendra Malik
  8. Andrew Zisserman

In Radiology, Nuclear Medicine and Imaging

  1. Jiashi Feng
  2. Ziwei Liu
  3. Alexei A. Efros
  4. Xintao Wang
  5. Ying Shan
  6. Ming-Hsuan Yang
  7. Jia-Bin Huang
  8. Yu Qiao

In Statistical and Nonlinear Physics

  1. Eli Shechtman
  2. Xintao Wang
  3. Ying Shan
  4. Richard Zhang
  5. Jun-Yan Zhu
  6. Alexei A. Efros
  7. Allan Jabri
  8. Ming-Hsuan Yang

In Generative Models

  1. Eli Shechtman
  2. Jun-Yan Zhu
  3. Richard Zhang
  4. Ming-Hsuan Yang
  5. Alexei A. Efros
  6. Chen Change Loy
  7. Yujun Shen
  8. Ming-Yu Liu

In Multimodal Learning

  1. Jianfeng Gao
  2. Linjie Li
  3. Zhe Gan
  4. Jianwei Yang
  5. Pengchuan Zhang
  6. Lei Zhang
  7. Yong Jae Lee
  8. Steven C. H. Hoi

Citations in the corpus