Training-Free Layout Control with Cross-Attention Guidance
By Minghao Chen, Iro Laina, Andrea Vedaldi. IEEE Workshop/Winter Conference on Applications of Computer Vision, 2023
373 citations recorded by the source corpus.
Recent diffusion-based generators can produce high-quality images from textual prompts. However, they often disregard textual instructions that specify the spatial layout of the composition. We propose a simple approach that achieves robust layout control without the need for training or fine-tuning of the image generator. Our technique manipulates the cross-attention layers that the model uses to interface textual and visual information and steers the generation in the desired direction given, e.g., a user-specified layout. To determine how to best guide attention, we study the role of attention maps and explore two alternative strategies, forward and backward guidance. We thoroughly evaluate our approach on three benchmarks and provide several qualitative examples and a comparative analysis of the two strategies that demonstrate the superiority of backward guidance compared to forward guidance, as well as prior work. We further demonstrate the versatility of layout guidance by extending it to applications such as editing the layout and context of real images.
Subfields: Mechanical Engineering
View Paper: https://www.semanticscholar.org/paper/6a195df2f7611aa75d5734b2efb32a408d2f8348
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Mechanical Engineering
Citations in the corpus
- BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion (2023, background): “Note that, there are many concurrent works [23, 7, 3, 21] that studied a similar area.”
- ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (2024, methodology): “Our method enhances the base model’s ability to follow prompts and can be seamlessly integrated with training-free methods [13,20,58].”
- T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation (2023, neutral): “Previous work focused on concept conjunction and negation [7], attribute binding with colors [8, 9, 34], and spatial relationships between objects [35, 10].”
- Diffusion Self-Guidance for Controllable Image Generation (2023, methodology): “…not just the probabilities from a classifier. g could be the approximate energy from another model [17], a similarity score from a CLIP model [23], an arbitrary time-independent energy as in universal guidance [1], bounding box penalties on attention [6], or any attributes of the noisy images.”
- LayoutGPT: Compositional Visual Planning and Generation with Large Language Models (2023, methodology): “We feed the same generated layouts from LayoutGPT to LayoutGuidance [6] and compute image-level metrics.”