Image Generation From Layout

By Bo Zhao, Lili Meng, Weidong Yin, Leonid Sigal. 2018

241 citations recorded by the source corpus.

Despite significant recent progress on generative models, controlled generation of images depicting multiple and complex object layouts is still a difficult problem. Among the core challenges are the diversity of appearance a given object may possess and, as a result, exponential set of images consistent with a specified layout. To address these challenges, we propose a novel approach for layout-based image generation; we call it Layout2Im. Given the coarse spatial layout (bounding boxes + object categories), our model can generate a set of realistic images which have the correct objects in the desired locations. The representation of each object is disentangled into a specified/certain part (category) and an unspecified/uncertain part (appearance). The category is encoded using a word embedding and the appearance is distilled into a low-dimensional vector sampled from a normal distribution. Individual object representations are composed together using convolutional LSTM, to obtain an encoding of the complete layout, and then decoded to an image. Several loss terms are introduced to encourage accurate and diverse generation. The proposed Layout2Im model significantly outperforms the previous state of the art, boosting the best reported inception score by 24.66% and 28.57% on the very challenging COCO-Stuff and Visual Genome datasets, respectively. Extensive experiments also demonstrate our method’s ability to generate complex and diverse images with multiple objects.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Object-Centric Image Generation from Layouts (2020, methodology): “In parallel, other approaches have focused on generating directly from coarse layouts (Sun and Wu 2019; Zhao et al. 2019).”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, background): “An other widely-used form is image-to-image translation [16, 20,22,23,24,28,32,54,56,57], where both input and out put are images.”
GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, methodology): “, text [65, 70, 80], box [61, 62, 78], semantic masks [36, 47], images [8,38,83].”
Auto-Encoding Scene Graphs for Image Captioning (2018, background): “Thus, the scene graph can provide a beneficial prior for other vision tasks like VQA [40, 43],image generation [13, 59], and visual grounding [28].”
Image Synthesis From Reconfigurable Layout and Style (2019, methodology): “In comparison, the state-of-the-art methods include the very recent Layout2Im method [38], the scene graph to image (sg2im) method [16] and the pix2pix method [15].”
Specifying Object Attributes and Relations in Interactive Scene Generation (2019, methodology): “The results of [27] are from their paper.”
Modeling Image Composition for Complex Scene Generation (2022, background): “Compared with other conditions for complex scene generation, including textual descriptions [27], scene graphs [13, 23], and segmentation masks [20], layouts are much more user-friendly, controllable and flexible [45].”
Training-Free Layout Control with Cross-Attention Guidance (2023, background): “Closely related to achieving layout control is image generation with spatial conditioning, which is typically specified as labeled bounding boxes or semantic maps [36, 37, 41, 49, 5, 25].”
On the Evaluation of Conditional GANs (2019, background): “In order to overcome this shortcoming, researchers have resorted to reporting conditional consistency metrics in conjunction with FID [54, 29].”
Semantic Object Accuracy for Generative Text-to-Image Synthesis (2019, methodology): “Given a coarse image layout (bounding boxes and object labels) [31] generate images by disentangling each object into a speciﬁed part (e.g. object label) and unspeciﬁed part (appearance).”
PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph (2019, methodology): “Diversity Score doesn’t decrease because the random crops preserve the complexity accidentally.”
Image Synthesis from Layout with Locality-Aware Mask Adaption (2021, methodology): “The layout-to-image generation task is to convert the bounding boxes to a photorealistic image without segmentation annotation [37].”
LayoutDiffusion: Controllable Diffusion Model for Layout-to-Image Generation (2023, methodology): “The first image generation directly from the layout appears in Layout2Im [56] and is defined as a set of objects annotated with category and bbox.”
Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models (2023, methodology): “To provide users with more control over the synthesis process, several works employ a segmentation map or spatial conditioning [4, 17, 54].”
RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval (2020, methodology): “The COCO-Stuff [2] and Visual Genome [18] datasets are standard benchmark datasets for evaluating scene generation models [14,41,44].”
Few-shot Semantic Image Synthesis Using StyleGAN Prior (2021, unknown): “The input semantic masks are the same as those used in Figure 7. ity [6, 22, 23, 32, 31], generate multi-modal outputs [23, 18, 44, 10], and simplify input annotations using bounding boxes [41, 30, 19].”
Continuous Graph Flow (2019, background): “Layout generation from scene graphs is a crucial task in computer vision and bridges the gap between the symbolic graph-based scene description and the object layouts in the scene (Johnson et al., 2018; Zhao et al., 2019; Jyothi et al., 2019).”
BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion (2023, background): “Layout-to-image literature [19, 32, 33, 36, 40, 12, 2, 1] has studied on the way to synthesize images adhering to the spatial conditioning input.”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “Similarly, Layout2Im (Layoutbased Image generation) [153] is also an end-to-end model”