Modeling Image Composition for Complex Scene Generation

By Zuopeng Yang, Daqing Liu, Chaoyue Wang, Jie Yang, Dacheng Tao. 2022

64 citations recorded by the source corpus.

We present a method that achieves state-of-the-art results on challenging (few-shot) layout-to-image generation tasks by accurately modeling textures, structures and relationships contained in a complex scene. After compressing RGB images into patch tokens, we propose the Transformer with Focal Attention (TwFA) for exploring dependencies of object-to-object, object-to-patch and patch-to-patch. Compared to existing CNN-based and Transformer-based generation models that entangled modeling on pixel-level&patch-level and object-level&patch-level respectively, the proposed focal attention predicts the current patch token by only focusing on its highly-related tokens that specified by the spatial layout, thereby achieving disambiguation during training. Furthermore, the proposed TwFA largely increases the data efficiency during training, therefore we propose the first few-shot complex scene generation strategy based on the well-trained TwFA. Comprehensive experiments show the superiority of our method, which significantly increases both quantitative metrics and qualitative visual realism with respect to state-of-the-art CNN-based and transformer-based methods. Code is available at https://github.com/JohnDreamer/TwFA.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis (2022, background): “Conventionally, we need spatial information such as coordinates (Yang et al., 2022) as input to map their semantics into corresponding images.”
BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion (2023, methodology): “Layout-to-image literature [19, 32, 33, 36, 40, 12, 2, 1] has studied on the way to synthesize images adhering to the spatial conditioning input.”
Training-Free Layout Control with Cross-Attention Guidance (2023, background): “Closely related to achieving layout control is image generation with spatial conditioning, which is typically specified as labeled bounding boxes or semantic maps [36, 37, 41, 49, 5, 25].”
Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis (2022, methodology): “In More Experimental Results for Layout-to-Image Generation Continuing from the Section 4.2, we follow TwFA (Yang et al. 2022)”