Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis

By Wanshu Fan, Yen-Chun Chen, Dongdong Chen, Yu Cheng, Lu Yuan, Yu-Chiang Frank Wang. 2022

123 citations recorded by the source corpus.

Diffusion models (DMs) have shown great potential for high-quality image synthesis. However, when it comes to producing images with complex scenes, how to properly describe both image global structures and object details remains a challenging task. In this paper, we present Frido, a Feature Pyramid Diffusion model performing a multi-scale coarse-to-fine denoising process for image synthesis. Our model decomposes an input image into scale-dependent vector quantized features, followed by a coarse-to-fine gating for producing image output. During the above multi-scale representation learning stage, additional input conditions like text, scene graph, or image layout can be further exploited. Thus, Frido can be also applied for conditional or cross-modality image synthesis. We conduct extensive experiments over various unconditioned and conditional image generation tasks, ranging from text-to-image synthesis, layout-to-image, scene-graph-to-image, to label-to-image. More specifically, we achieved state-of-the-art FID scores on five benchmarks, namely layout-to-image on COCO and OpenImages, scene-graph-to-image on COCO and Visual Genome, and label-to-image on COCO.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services (2023, unknown): “AI models in mobile networks, AIGC can produce a vast array of content, including text, images, and audio, which can be seamlessly adjusted to suit evolving network demands and user preferences [120].”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Multi-Modal Learning Text-to-Image Generation [7],[186],[201],[167],[83],[62],[199],[116],[235] Text-to-Audio Generation [180],[261],[249], [136],[228],[99],[119] Robust Learning Robust Learning [168],[270],[16],[240],[248],[227]”
NExT-GPT: Any-to-Any Multimodal LLM (2023, background): “1 Introduction Recently, the topic of Artificial Intelligence Generated Content (AIGC) has witnessed unprecedented advancements with certain technologies, such as ChatGPT for text generation [59] and diffusion models for visual generation [21].”
Training-Free Layout Control with Cross-Attention Guidance (2023, background): “Closely related to achieving layout control is image generation with spatial conditioning, which is typically specified as labeled bounding boxes or semantic maps [36, 37, 41, 49, 5, 25].”
DiffusionDet: Diffusion Model for Object Detection (2022, background): “Diffusion models have recently demonstrated remarkable results in fields including computer vision [4,21,33,35,39,68,71,75,76,82,104,108], nature language processing [3,30,52], audio processing [41,48,50,70, 91,100,103], graph-related topics [42], interdisciplinary applications…”