Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

By Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, Daniel Cohen-Or. ACM Transactions on Graphics, 2023

864 citations recorded by the source corpus.

Recent text-to-image generative models have demonstrated an unparalleled ability to generate diverse and creative imagery guided by a target text prompt. While revolutionary, current state-of-the-art diffusion models may still fail in generating images that fully convey the semantics in the given text prompt. We analyze the publicly available Stable Diffusion model and assess the existence of catastrophic neglect, where the model fails to generate one or more of the subjects from the input prompt. Moreover, we find that in some cases the model also fails to correctly bind attributes (e.g., colors) to their corresponding subjects. To help mitigate these failure cases, we introduce the concept of Generative Semantic Nursing (GSN), where we seek to intervene in the generative process on the fly during inference time to improve the faithfulness of the generated images. Using an attention-based formulation of GSN, dubbed Attend-and-Excite, we guide the model to refine the cross-attention units to attend to all subject tokens in the text prompt and strengthen --- or excite --- their activations, encouraging the model to generate all subjects described in the text prompt. We compare our approach to alternative approaches and demonstrate that it conveys the desired concepts more faithfully across a range of text prompts. Code is available at our project page: https://attendandexcite.github.io/Attend-and-Excite/.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c3c7464acb90049c5f520b0732dc7435ba3690bd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey on Generative Diffusion Models (2022, background): “[141] enforces the cross-attention to attend to all subject tokens in the text prompt and enlarge their activations, encouraging the model to faithfully generate all subjects described in the text prompt.”
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models (2023, methodology): “Direct multi-object generation often encounters issues such as missing objects and attribute binding [5, 6].”
TokenFlow: Consistent Diffusion Features for Consistent Video Editing (2023, background): “Recently, a surge of works demonstrated how text-to-image diffusion models can be readily adapted to various editing and generation tasks, by performing simple operations on the intermediate feature representation of the diffusion network [6, 16, 28, 51, 12, 32, 4].”
MACE: Mass Concept Erasure in Diffusion Models (2024, background): “Intuitively, if a concept is to appear in generated images, it should exert signiﬁcant inﬂuence on several patches of those images [10, 46].”
Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models (2023, positive): “Moreover, recent controllable text-to-image synthesis works [8, 20] have highlighted that cross attention conditioning is a pivotal factor in determining the primary concepts featured in generated images.”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, methodology): “This phenomenon is not unique to D REAM LLM, but is also observed in specialized compositional generation methodologies, such as StructureDiffusion (Feng et al., 2023; Chefer et al., 2023).”
T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation (2023, positive): “Comparisons A similar phenomenon was also observed in previous work [9] that Composable Diffusion often generates images containing a mixture of the subjects.”
FastComposer: Tuning-Free Multi-subject Image Generation with Localized Attention (2023, neutral): “Motivated by [5, 12], we apply the localization loss to the downsampled cross-attention maps, i.”
ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (2024, background): “5, v2 and XL [40] model, Composable v2 [35], Structured v2 [19], Attn-Exct v2 [11], GORS [27], PixArt-α [12] and DALL-E 2 [45].”
MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing (2023, methodology): “Unlike previous works [10, 21, 3] that usually operate on cross-attention in T2I models, we propose to convert existing self-attention to mutual self-attention, so that it can query correlated local structures and textures from a source image for consistency.”
Training-Free Layout Control with Cross-Attention Guidance (2023, background): “Others focus on improving compositionality, e.g ., by combining multiple diffusion models with different operators [30], and attribute binding [5,13].”
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models (2023, methodology): “Attend-and-Excite [4] optimizes attention regions so that objects attend on separate regions.”
BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion (2023, methodology): “[10] and recently introduced [6] operated on the cross-attention for better content consistency to subjects in text prompts.”