BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion

By Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, Mike Zheng Shou. IEEE International Conference on Computer Vision, 2023

332 citations recorded by the source corpus.

Recent text-to-image diffusion models have demonstrated an astonishing capacity to generate high-quality images. However, researchers mainly studied the way of synthesizing images with only text prompts. While some works have explored using other modalities as conditions, considerable paired data, e.g., box/mask-image pairs, and fine-tuning time are required for nurturing models. As such paired data is time-consuming and labor-intensive to acquire and restricted to a closed set, this potentially becomes the bottleneck for applications in an open world. This paper focuses on the simplest form of user-provided conditions, e.g., box or scribble. To mitigate the aforementioned problem, we propose a training-free method to control objects and contexts in the synthesized images adhering to the given spatial conditions. Specifically, three spatial constraints, i.e., Inner-Box, Outer-Box, and Corner Constraints, are designed and seamlessly integrated into the denoising step of diffusion models, requiring no additional training and massive annotated layout data. Extensive experimental results demonstrate that the proposed constraints can control what and where to present in the images while retaining the ability of Diffusion models to synthesize with high fidelity and diverse concept coverage.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/5fe41b0acee32c041f01d97c6bb579fedcb7950d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Training-Free Layout Control with Cross-Attention Guidance (2023, methodology): “The images under the line are generated by our method on top of Dreambooth and TI. to evaluate layout control, since it contains image-caption pairs with visual grounding.”
ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (2024, background): “Some works [6, 11, 13, 20, 28, 31, 35, 44, 56] attempt to manipulate cross-attention maps or latents according to spatial or semantic constraints in the prompt.”