High-Resolution Complex Scene Synthesis with Transformers

By Manuel Jahn, Robin Rombach, Björn Ommer. 2021

44 citations recorded by the source corpus.

The use of coarse-grained layouts for controllable synthesis of complex scene images via deep generative models has recently gained popularity. However, results of current approaches still fall short of their promise of high-resolution synthesis. We hypothesize that this is mostly due to the highly engineered nature of these approaches which often rely on auxiliary losses and intermediate steps such as mask generators. In this note, we present an orthogonal approach to this task, where the generative model is based on pure likelihood training without additional objectives. To do so, we first optimize a powerful compression model with adversarial training which learns to reconstruct its inputs via a discrete latent bottleneck and thereby effectively strips the latent representation of high-frequency details such as texture. Subsequently, we train an autoregressive transformer model to learn the distribution of the discrete image representations conditioned on a tokenized version of the layouts. Our experiments show that the resulting system is able to synthesize high-quality images consistent with the given layouts. In particular, we improve the state-of-the-art FID score on COCO-Stuff and on Visual Genome by up to 19% and 53% and demonstrate the synthesis of images up to 512 x 512 px on COCO and Open Images.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “Our OpenImages model surpasses the results of Jahn et al [33] by a margin of nearly 11 in terms of FID.”
Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis (2022, methodology): “As for the OpenImage dataset, we follow the setting of VQGAN+T (Jahn, Rombach, and Ommer 2021) and sample 2,048 images in the validation set for evaluation.”
Modeling Image Composition for Complex Scene Generation (2022, methodology): “In contrast, the Transformer-based methods tokenize the layout into object tokens and employ a pre-trained compression model to quantize the image into a sequence of discrete patch tokens, thus simplifying the image generation task as an image patch composition task implemented by a Transformer.”