Semantic Bottleneck Scene Generation

By Samaneh Azadi, Michael Tschannen, Eric Tzeng, Sylvain Gelly, Trevor Darrell, Mario Lucic. 2019

31 citations recorded by the source corpus.

Coupling the high-fidelity generation capabilities of label-conditional image synthesis methods with the flexibility of unconditional generative models, we propose a semantic bottleneck GAN model for unconditional synthesis of complex scenes. We assume pixel-wise segmentation labels are available during training and use them to learn the scene structure. During inference, our model first synthesizes a realistic segmentation layout from scratch, then synthesizes a realistic scene conditioned on that layout. For the former, we use an unconditional progressive segmentation generation network that captures the distribution of realistic semantic scene layouts. For the latter, we use a conditional segmentation-to-image synthesis network that captures the distribution of photo-realistic images conditioned on the semantic layout. When trained end-to-end, the resulting model outperforms state-of-the-art generative models in unsupervised image synthesis on two challenging domains in terms of the Frechet Inception Distance and user-study evaluations. Moreover, we demonstrate the generated segmentation maps can be used as additional training data to strongly improve recent segmentation-to-image synthesis networks.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

1. Andreas Geiger
2. Axel Sauer
3. Jens Muller
4. Kashyap Chitta
5. Drew A. Hudson
6. C. Lawrence Zitnick
7. Michael Niemeyer
8. Songyou Peng

In Computer Vision and Pattern Recognition

1. Andreas Geiger
2. Luc Van Gool
3. Dacheng Tao
4. Trevor Darrell
5. Xiaogang Wang
6. Jitendra Malik
7. Ming-Hsuan Yang
8. Andrew Zisserman

In Generative Models

1. Axel Sauer
2. Andreas Geiger
3. Kashyap Chitta
4. Jens Muller
5. Drew A. Hudson
6. C. Lawrence Zitnick
7. Tero Karras
8. Timo Aila

Citations in the corpus

• Projected GANs Converge Faster (2021, methodology): “We compare on LSUN cat and horse [84], ADE indoor (a subset of ADE [91] proposed in [3]), the full Oxford flowers dataset with 8k images [56], KITTI-fisheye (a subset of KITTI-360 [46], consisting of fisheye images), STL-10 [8], CUB200 [80], Stanford Dogs [36], and Stanford Cars [42].”
• Generative Adversarial Transformers (2021, background): “But while automatically produced images for faces, single objects or natural scenery have reached astonishing fidelity, becoming nearly indistinguishable from real samples, the unconditional synthesis of more structured or compositional scenes is still lagging behind, suffering from inferior coheren”