Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis

By Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, X. Wang, William Yang Wang. International Conference on Learning Representations, 2022

441 citations recorded by the source corpus.

Large-scale diffusion models have achieved state-of-the-art results on text-to-image synthesis (T2I) tasks. Despite their ability to generate high-quality yet creative images, we observe that attribution-binding and compositional capabilities are still considered major challenging issues, especially when involving multiple objects. In this work, we improve the compositional skills of T2I models, specifically more accurate attribute binding and better image compositions. To do this, we incorporate linguistic structures with the diffusion guidance process based on the controllable properties of manipulating cross-attention layers in diffusion-based T2I models. We observe that keys and values in cross-attention layers have strong semantic meanings associated with object layouts and content. Therefore, we can better preserve the compositional semantics in the generated image by manipulating the cross-attention representations based on linguistic insights. Built upon Stable Diffusion, a SOTA T2I model, our structured cross-attention design is efficient that requires no additional training samples. We achieve better compositional skills in qualitative and quantitative results, leading to a 5-8% advantage in head-to-head user comparison studies. Lastly, we conduct an in-depth analysis to reveal potential causes of incorrect image compositions and justify the properties of cross-attention layers in the generation process.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/25de00096c45121a06668bc501f91adec5d0aff9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment (2023, background): “This supports prior claims that spatial reasoning and attribute binding are difficult for T2I models [9, 13, 14].”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, methodology): “This phenomenon is not unique to D REAM LLM, but is also observed in specialized compositional generation methodologies, such as StructureDiffusion (Feng et al., 2023; Chefer et al., 2023).”
Aligning Text-to-Image Models using Human Feedback (2023, background): “However, major challenges remain in domains where large-scale text-to-image models fail to generate images that are well-aligned with text prompts (Feng et al., 2022; Liu et al., 2022a;b).”
T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation (2023, unknown): “The 1,000 text prompts related to color binding are constructed with 480 prompts from CC500 [8], 200 prompts from COCO [38], and 320 prompts generated by ChatGPT.”
Human Preference Score: Better Aligning Text-to-image Models with Human Preference (2023, methodology): “[8] propose a training-free method to guide diffusion models for better compositional capabilities.”
TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering (2023, background): “models often fail to compose multiple objects [40, 11, 35], bind attributes to the wrong objects [11], and struggle in generating visual text [36].”
Exploiting Diffusion Prior for Real-World Image Super-Resolution (2023, unknown): “2.”
ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation (2023, background): “The inherent discrepancy leads to several well-documented issues in the generated images [14, 28], including but not limited to: • Text-image Alignment: failing to accurately depict all the numbers, attributes, properties, and relationships of objects described in text prompts, as shown in Figure 1 ”
T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models (2023, methodology): “Some methods [11, 9, 1] are also proposed to modulate the cross-attention maps in pre-trained T2I models to guide the generation process.”
Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models (2023, methodology): “[2022] propose StructureDiffusion which employs consistency trees or scene graphs to split the prompt into several noun phrases.”
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models (2023, background): “For instance, text-to-image generation (T2I) models suffer from generating objects with specified counts, positions, and attributes [10].”
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (2023, unknown): “…cause of this may lie in the used pretrained text-encoders: ﬁrstly, they are trained to compress all information into a single token, so they may fail at binding only the right attributes and objects, Feng et al. [8] mitigate this issue by explicitly encoding word relationships into the encoding.”
FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model (2023, unknown): “[27, 11, 14] attempt to use the cross-attention control to realize the conditional generation; [5, 6, 7, 8, 19, 24, 39, 41, 44, 43] directly modify the intermediate results to achieve zero-shot image restoration; [25] realizes image translation by adjusting the initial noisy images.”
NExT-GPT: Any-to-Any Multimodal LLM (2023, methodology): “Moreover, to generate high-quality content, a multitude of strong-performing methods have been proposed, such as Transformer [82, 101, 17, 24], GANs [53, 7, 93, 110], VAEs [81, 67], Flow models [73, 6] and the current state-of-the-art diffusion models [31, 64, 57, 22, 68].”
Training-Free Layout Control with Cross-Attention Guidance (2023, background): “Others focus on improving compositionality, e.g ., by combining multiple diffusion models with different operators [30], and attribute binding [5,13].”
DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models (2023, background): “However, there are still many known weaknesses of existing text-to-image models, such as compositionality and attribute binding [6, 20] or text rendering [21].”
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models (2023, methodology): “Direct multi-concept sampling often encounters challenges of missing objects and attribute binding [6, 31, 32, 33, 34].”
BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion (2023, methodology): “[10] and recently introduced [6] operated on the cross-attention for better content consistency to subjects in text prompts.”
ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (2024, neutral): “5, v2 and XL [40] model, Composable v2 [35], Structured v2 [19], Attn-Exct v2 [11], GORS [27], PixArt-α [12] and DALL-E 2 [45].”