T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models

By Chong Mou, Xintao Wang, Liangbin Xie, Jing Zhang, Zhongang Qi, Ying Shan, Xiaohu Qie. AAAI Conference on Artificial Intelligence, 2023

1,776 citations recorded by the source corpus.

The incredible generative ability of large-scale text-to-image (T2I) models has demonstrated strong power of learning complex structures and meaningful semantics. However, relying solely on text prompts cannot fully take advantage of the knowledge learned by the model, especially when flexible and accurate controlling (e.g., structure and color) is needed. In this paper, we aim to ``dig out" the capabilities that T2I models have implicitly learned, and then explicitly use them to control the generation more granularly. Specifically, we propose to learn low-cost T2I-Adapters to align internal knowledge in T2I models with external control signals, while freezing the original large T2I models. In this way, we can train various adapters according to different conditions, achieving rich control and editing effects in the color and structure of the generation results. Further, the proposed T2I-Adapters have attractive properties of practical value, such as composability and generalization ability. Extensive experiments demonstrate that our T2I-Adapter has promising generation quality and a wide range of applications. Our code is available at https://github.com/TencentARC/T2I-Adapter.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/58842cdca3ea68f7b9e638b288fc247a6f26dafc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks (2024, background): “…16, 28], referring object detection and segmentation [41, 37, 86], object tracking [67, 84], image generation [75, 54, 48, 45, 23, 14, 52, 22, 82, 44], image editing [42, 1, 2, 53, 21], human-centric perception and understanding [72, 71, 73, 70, 69, 4, 33, 74], and human-centric motion…”
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding (2023, positive): “The remarkable performance of these models can be attributable to high-quality large-scale text-image datasets [7, 60, 61], the continuous upgrades of foundational models [8, 48], conditioning encoders [27, 50, 51], and the improvement of controllability [36, 45, 73, 75].”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, methodology): “To enable conditional synthesis with ﬂexible input, ControlNet [72] and T2I-Adapter [34] incorporate a speciﬁc spatial condition into the model, providing more ﬁne-grained control.”
AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models (2023, neutral): “Supervised Training Another type of methods [47, 6, 54, 37] handle editing tasks in a supervised manner.”
DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior (2023, methodology): “One might use non-uniform sampling to increase the probability of optimization in the early sampling stage and achieves better color controlling [42].”
SDDGR: Stable Diffusion-Based Deep Generative Replay for Class Incremental Object Detection (2024, background): “[21, 32, 41, 65, 70] have gained popularity for controlled generation by incorporating additional conditions.”
SVDiff: Compact Parameter Space for Diffusion Fine-Tuning (2023, background): “Fine-tuning generative models for personalization Recent works have focused on customizing and personalizing text-to-image diffusion models by fine-tuning the text embedding [16], full weights [52], cross-attention layers [32], or adapters [77, 38] using a few personalized images.”
ControlVideo: Training-free Controllable Text-to-Video Generation (2023, neutral): “Besides text prompts, several works [20, 43] also introduce additional structure conditions to pre-trained text-to-image diffusion models for controllable text-to-image generation.”
IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models (2023, methodology): “Almost concurrently, T2I-adapter [11] employs a simple and lightweight adapter to achieve fine-grained control in the color and structure of the generated images.”
Effective Whole-body Pose Estimation with Two-stages Distillation (2023, background): “Recently, controllable image generation [12, 37, 39, 63, 33, 20] has witnessed signiﬁcant advancements.”
InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning (2023, background): “More recently, approaches built upon pre-trained text-to-image models have been proposed for more controllable multi-modal image generation [20, 23, 42].”
DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors (2023, methodology): “Similar to adding controls in T2I [48, 61, 88, 93], introducing control signals in T2V, such as structure [17, 83], pose [44, 94], and Canny edge [39], has been increasingly receiving much attention.”
Pix2Video: Video Editing using Image Diffusion (2023, positive): “These methods work by either finetuning [55] an existing model, introducing adapter layers [33] or other trainable modules [54, 63], or utilizing an ensemble of denoising networks [2].”
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation (2022, positive): “Notably, our method is inherently compatible with exiting personalized and conditional pretrained T2I models, such as Dream-Booth [40] and T2I-Adapter [30], providing a personalized and controllable user interface.”
Step1X-Edit: A Practical Framework for General Image Editing (2025, positive): “Based on the above text2image models, ControlNet [67], and T2I-Adapter [36] inject spatial or task-specific control into the generation process.”
Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance (2024, background): “Moreover, in the endeavor to enhance control over visual generation, recent studies such as ControlNet [53], T2I-Adapter [25], and IP-Adapter [49] have delved into the incorporation of supplementary encoder layers.”
Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, background): “ControlNet[60] and T2I-Adapter[27] delve into the control-lability of visual generation by incorporating additional encoding layers, facilitating controlled generation under various conditions such as pose, mask, edge and depth.”
OmniGen: Unified Image Generation (2024, positive): “Many efforts have been made to extend the capabilities of diffusion models, such as ControlNet [75], T2I-Adapter [43], StyleShot [15].”
ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis (2024, background): “Control-Net [44], T2I-adapter [45], and GLIGEN [46] pioneered the introduction of condition signals for T2I generation.”