Adding Conditional Control to Text-to-Image Diffusion Models
By Lvmin Zhang, Anyi Rao, Maneesh Agrawala. 2023
7,354 citations recorded by the source corpus.
We present ControlNet, a neural network architecture to add spatial conditioning controls to large, pretrained text-to-image diffusion models. ControlNet locks the production-ready large diffusion models, and reuses their deep and robust encoding layers pretrained with billions of images as a strong backbone to learn a diverse set of conditional controls. The neural architecture is connected with "zero convolutions" (zero-initialized convolution layers) that progressively grow the parameters from zero and ensure that no harmful noise could affect the finetuning. We test various conditioning controls, e.g., edges, depth, segmentation, human pose, etc., with Stable Diffusion, using single or multiple conditions, with or without prompts. We show that the training of ControlNets is robust with small (<50k) and large (>1m) datasets. Extensive results show that ControlNet may facilitate wider applications to control image diffusion models.
Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Statistical and Nonlinear Physics, Generative Models, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Chen Change Loy
- Jitendra Malik
- Andrew Zisserman
In Radiology, Nuclear Medicine and Imaging
In Statistical and Nonlinear Physics
- Eli Shechtman
- Xintao Wang
- Ying Shan
- Richard Zhang
- Jun-Yan Zhu
- Alexei A. Efros
- Allan Jabri
- Ming-Hsuan Yang
In Generative Models
- Eli Shechtman
- Jun-Yan Zhu
- Richard Zhang
- Ming-Hsuan Yang
- Alexei A. Efros
- Chen Change Loy
- Yujun Shen
- Ming-Yu Liu
In Multimodal Learning
Citations in the corpus
- Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, methodology): “We collect foundation models from Hug-gingFace Transformers [43] 2 , Maskformer [10] 3 and Con-trolNet [53] 4 .”
- DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, methodology): “Other methods attempt to learn from distillation, such as Alpaca (Taori et al., 2023) and Vicuna (Chiang et al., 2023).”
- GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction (2023, background): “For instance, ControlNet [45] needs the image path saved conditions (e.g., the pose map, depth map, or segment map) and the input text described user commands.”
- BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion (2023, background): “Beyond text prompts as conditions, Stable Diffusion [27] and ControlNet [38] have also studied other modalities as conditioning input and provided qualitative results.”
- Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization (2023, methodology): “2021) can generate highquality natural images, Zhang and Agrawala (Zhang and Agrawala 2023) proposed ControlNet, which enables conditional inputs like edge maps, segmentation maps, etc.”
- T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models (2023, background): “As concurrent works, [45] learns task-specific ControlNet to enable conditional generation for the pre-trained T2I model.”
- FateZero: Fusing Attentions for Zero-shot Text-based Video Editing (2023, background): “Thus, a framework based on image models can be more valuable than on video ones [36], thanks to the various open-sourced image models in the community [1,37,42,54].”
- OminiControl: Minimal and Universal Control for Diffusion Transformer (2024, neutral): “To address this limitation, some works have explored image-based control methods [2, 14, 15, 18, 26– 30, 37, 42, 46, 50, 52, 54, 56, 57, 57, 60, 61, 63], enabling users to specify their intentions through reference images or visual hints, offering more precise guidance than text alone [44].”
- Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation (2023, methodology): “To improve spatial controllability, [45] introduce a side path called ControlNet to Stable Diffusion to accept extra conditions like edges, depth and human pose.”
- InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (2023, methodology): “ControlNet [43] extends the pre-trained model to take multi-modal data as inputs, but it does not show evidence of identity preservation ability when conditioned on images.”
- A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “Furthermore, Qi et al. (2016) introduced models that process 3D data, including point clouds, for object classification tasks, employing CNNs to learn representations from volumetric data and multiple 2D views of 3D objects [137].”
- Segment Everything Everywhere All at Once (2023, background): “In the image generation area, several recent works attempt to combine text prompts with other types like sketches or layouts, to capture user intents more precisely [42, 39, 58], compose novel prompts [62, 27], and support multi-round human-AI interactions [1, 49].”
- VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, methodology): “To enable conditional synthesis with flexible input, ControlNet [72] and T2I-Adapter [34] incorporate a specific spatial condition into the model, providing more fine-grained control.”
- SVDiff: Compact Parameter Space for Diffusion Fine-Tuning (2023, background): “Fine-tuning generative models for personalization Recent works have focused on customizing and personalizing text-to-image diffusion models by fine-tuning the text embedding [16], full weights [52], cross-attention layers [32], or adapters [77, 38] using a few personalized images.”
- Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model (2023, methodology): “In addition to the base Zero123++ model, we also release a depth-controlled version of Zero123++ built with ControlNet [25].”
- Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos (2023, methodology): “Our method can also be used in other related conditional video generation tasks as shown in concurrent conditional image generation works [41, 20].”