Composer: Creative and Controllable Image Synthesis with Composable Conditions

By Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, Jingren Zhou. International Conference on Machine Learning, 2023

394 citations recorded by the source corpus.

Recent large-scale generative models learned on big data are capable of synthesizing incredible images yet suffer from limited controllability. This work offers a new generation paradigm that allows flexible control of the output image, such as spatial layout and palette, while maintaining the synthesis quality and model creativity. With compositionality as the core idea, we first decompose an image into representative factors, and then train a diffusion model with all these factors as the conditions to recompose the input. At the inference stage, the rich intermediate representations work as composable elements, leading to a huge design space (i.e., exponentially proportional to the number of decomposed factors) for customizable content creation. It is noteworthy that our approach, which we call Composer, supports various levels of conditions, such as text description as the global information, depth map and sketch as the local guidance, color histogram for low-level details, etc. Besides improving controllability, we confirm that Composer serves as a general framework and facilitates a wide range of classical generative tasks without retraining. Code and models will be made available.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/26e5b933b8f60bd749d428b5ff813b2abcd765d8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, background): “Recently, benefiting from advancements in text-to-image research[2, 19, 29, 32, 34, 36], video generation (e.g., text-to-video, video editing)[4, 10, 12, 15–17, 23, 30, 40, 48, 52] has also achieved notable progress in terms of visual quality and diversity.”
IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models (2023, methodology): “To achieve controllable image synthesis, Composer [27] presents a joint fine-tuning strategy with various conditions on a pretrained diffusion model conditioned on image embedding.”
T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models (2023, background): “[14] proposed to retrain a diffusion model conditioned on a set of control factors.”
Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models (2023, methodology): “Broadly speaking, there are two strategies for implementing controllable diffusion models: training from scratch [19] and fine-tuning lightweight adapters [20, 21] on frozen pretrained T2I diffusion models.”
SDDGR: Stable Diffusion-Based Deep Generative Replay for Class Incremental Object Detection (2024, background): “[21, 32, 41, 65, 70] have gained popularity for controlled generation by incorporating additional conditions.”
AnyDoor: Zero-shot Object-level Image Customization (2023, neutral): “Image generation is flourishing with the booming advancement of diffusion models [22, 38, 41, 42, 44, 60].”
I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models (2023, background): “The initial efforts in image generation utilizing DPMs were primarily centered around enhancing performance [7, 20, 22].”
Training-Free Layout Control with Cross-Attention Guidance (2023, background): “Finally, building on top of diffusion, some recent works show controllable image generation from various conditioning signals [1, 48, 16].”
Adding Conditional Control to Text-to-Image Diffusion Models (2023, background): “Concurrent works [8, 9, 32, 56] examine a wide variety of ways to control diffusion models.”
Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance (2024, background): “Diffusion-based models [3,17,27, 29, 32,34] have rapidly emerged as a fundamental component in the domain of text-to-image generation, renowned for their capacity to yield highly promising generative outcomes.”
InstantID: Zero-shot Identity-Preserving Generation in Seconds (2024, background): “Text-to-image diffusion models [15,19,16,12,1,7] achieve state-of-the-art image generation results and have received unprecedented interest from the community in recent years.”
ModelScope Text-to-Video Technical Report (2023, methodology): “This encompasses textual content [40, 5, 56], visual content [39, 24, 48, 44, 27, 71, 16, 38, 51, 59] and auditory content [4, 32, 31].”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, methodology): “Expanding upon this idea, Composer [26] proposes a new generative paradigm centered on the concept of compositionality , which is capable of composing an image with various input conditions, leading to remarkable ﬂexibility.”
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models (2023, background): “While one could additionally integrate models that synthesize images with the guidance of layouts [56, 27, 2], keypoints [27], or sketches [21, 57], users still have to create fine-grained inputs on their own, resulting in extra efforts and degraded efficiency compared to using pure language instruc”
SVDiff: Compact Parameter Space for Diffusion Fine-Tuning (2023, background): “Text-to-image diffusion models Diffusion models [58, 61, 19, 62, 40, 59, 18, 60, 10] have proven to be highly effective in learning data distributions and have shown impressive results in image synthesis, leading to various applications [74, 45, 9, 49, 26, 23, 34, 56, 28, 24, 29].”