Compositional Visual Generation with Composable Diffusion Models

By Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, Josh Tenenbaum. 2022

742 citations recorded by the source corpus.

Large text-guided diffusion models, such as DALLE-2, are able to generate stunning photorealistic images given natural language descriptions. While such models are highly flexible, they struggle to understand the composition of certain concepts, such as confusing the attributes of different objects or relations between objects. In this paper, we propose an alternative structured approach for compositional generation using diffusion models. An image is generated by composing a set of diffusion models, with each of them modeling a certain component of the image. To do this, we interpret diffusion models as energy-based models in which the data distributions defined by the energy functions may be explicitly combined. The proposed method can generate scenes at test time that are substantially more complex than those seen in training, composing sentence descriptions, object relations, human facial attributes, and even generalizing to new combinations that are rarely seen in the real world. We further illustrate how our approach may be used to compose pre-trained text-guided diffusion models and generate photorealistic images containing all the details described in the input descriptions, including the binding of certain object attributes that have been shown difficult for DALLE-2. These results point to the effectiveness of the proposed method in promoting structured generalization for visual generation. Project page: https://energy-based-model.github.io/Compositional-Visual-Generation-with-Composable-Diffusion-Models/

Subfields: Computer Vision and Pattern Recognition, General Social Sciences, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In General Social Sciences

In Generative Models

Citations in the corpus

InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, methodology): “Liu et al. [37] demonstrate that a conditional diffusion model can compose score estimates from multiple different conditioning values.”
Composer: Creative and Controllable Image Synthesis with Composable Conditions (2023, background): “…et al., 2023), segmentation maps (Rom-bach et al., 2021; Wang et al., 2022b; Couairon et al., 2022), a composition of multiple text descriptions (Liu et al., 2022), and depthmaps (stability.ai, 2022), or by ﬁnetuning parameters on a few subject-speciﬁc data (Gal et al., 2022; Mokady et al.,…”
Training Diffusion Models with Reinforcement Learning (2023, positive): “for new concepts [12], composing models [11, 27], adapters for additional input constraints [58], and inference-time techniques such as classifier [10] and classifier-free [18] guidance.”
Diffusion Self-Guidance for Controllable Image Generation (2023, methodology): “From the score-based perspective, we can think of guidance as composing score functions to sample from richer distributions or to introduce conditioning on auxiliary information [7, 17, 35].”
Magic3D: High-Resolution Text-to-3D Content Creation (2022, neutral): “for compositional text-toimage generation [21].”
Composing Ensembles of Pre-trained Models via Iterative Consensus (2022, methodology): “Other works compose pre-trained models by composing learned probability distributions of the data, such as energy-based models (Liu et al., 2022; 2021; Du et al., 2020), which can be applied to image generation.”
Erasing Concepts from Diffusion Models (2023, background): “Our work is inspired by the observation [10, 11] that set-like composition can be performed naturally on energy-based models and diffusion counterparts [22] naturally via arithmetic on the score or the noise predictions.”
FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model (2023, background): “Conditional diffusion models (CDMs) with diverse conditions have been proposed, such as text [20, 1, 13, 23, 33, 37, 32, 35, 22, 29], class labels [10], degraded images [5, 6, 7, 8, 19, 24, 39, 41, 44, 34, 36, 45], segmentation maps [28, 49], landmarks [28, 49], hand-drawn sketches [28, 49], style…”
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis (2022, methodology): “Apart from Stable Diffusion, we also compare to Composable Diffusion (Liu et al., 2022) implemented on top of Stable Diffusion.”
Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models (2023, methodology): “Figure 23 presents additional results obtained using Composable Diffusion that demonstrate this phenomenon.”
ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation (2023, background): “The inherent discrepancy leads to several well-documented issues in the generated images [14, 28], including but not limited to: • Text-image Alignment: failing to accurately depict all the numbers, attributes, properties, and relationships of objects described in text prompts, as shown in Figure 1 ”
TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering (2023, methodology): “The text input is from the MSCOCO validation set. models often fail to compose multiple objects [40, 11, 35], bind attributes to the wrong objects [11], and struggle in generating visual text [36].”
Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation (2022, methodology): “To increase the deviation from guidance image content, we use negative prompting [26] with a prompt Pn that describes the guidance image.”
Aligning Text-to-Image Models using Human Feedback (2023, background): “However, major challenges remain in domains where large-scale text-to-image models fail to generate images that are well-aligned with text prompts (Feng et al., 2022; Liu et al., 2022a;b).”
Directly Fine-Tuning Diffusion Models on Differentiable Rewards (2023, neutral): “While it would also be possible to instead directly sum the score functions of the models (similar to Liu et al. 2022), this would increase generation time proportionally to the number of combined models.”
Improving Factuality and Reasoning in Language Models through Multiagent Debate (2023, background): “Our work is also related to existing works that focus on text generation by combining different models [4, 17, 32, 1, 5].”