Image Synthesis from Layout with Locality-Aware Mask Adaption

By Zejian Li, J. Wu, Immanuel Koh, Yongchuan Tang, Lingyun Sun. 2021

89 citations recorded by the source corpus.

This paper is concerned with synthesizing images conditioned on a layout (a set of bounding boxes with object categories). Existing works construct a layout-mask-image pipeline. Object masks are generated separately and mapped to bounding boxes to form a whole semantic segmentation mask (layout-to-mask), with which a new image is generated (mask-to-image). However, overlapped boxes in layouts result in overlapped object masks, which reduces the mask clarity and causes confusion in image generation. We hypothesize the importance of generating clean and semantically clear semantic masks. The hypothesis is supported by the finding that the performance of state-of-the-art LostGAN decreases when input masks are tainted. Motivated by this hypothesis, we propose Locality-Aware Mask Adaption (LAMA) module to adapt overlapped or nearby object masks in the generation. Experimental results show our proposed model with LAMA outperforms existing approaches regarding visual fidelity and alignment with input layouts. On COCO-stuff in 256×256, our method improves the state-of-the-art FID score from 41.65 to 31.12 and the SceneFID from 22.00 to 18.64.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

1. Ying Shan
2. Chaoyue Wang
3. Dacheng Tao
4. Lu Yuan
5. Mike Zheng Shou
6. Zuopeng Yang
7. Xi Li
8. Jie Yang

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Lu Yuan
3. Luc Van Gool
4. Ying Shan
5. Yu Cheng
6. Ming-Hsuan Yang
7. Dongdong Chen
8. Jianfeng Gao

In Generative Models

1. Ying Shan
2. Chaoyue Wang
3. Dacheng Tao
4. Yu Cheng
5. Lu Yuan
6. Mike Zheng Shou
7. Zuopeng Yang
8. Xi Li

Citations in the corpus

• Modeling Image Composition for Complex Scene Generation (2022, methodology): “Although CNN-based layout-to-image generation methods have achieved promising performance on texture synthesis, they may still suffer from accurately modeling the dependencies among pixels (or object parts), which hinders the model generated more realistic scene images.”
• BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion (2023, methodology): “Layout-to-image literature [19, 32, 33, 36, 40, 12, 2, 1] has studied on the way to synthesize images adhering to the spatial conditioning input.”
• Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis (2022, methodology): “In addition, we considered other task-specific metrics such as CLIP score (Hessel et al. 2021), Precision and Recall (Sajjadi et al. 2018), SceneFID (Sylvain et al. 2021), YOLO score (Li et al. 2021), PSNR (Hore and Ziou 2010), and SSIM (Wang et al. 2004) when applicable.”
• LayoutDiffusion: Controllable Diffusion Model for Layout-to-Image Generation (2023, methodology): “YOLOScore [20] evaluates 80 thing categories bbox mAP on generated images using a pretrained YOLOv4 [4] model, and shows the precision of control in one generated model.”