Photographic Image Synthesis with Cascaded Refinement Networks

By Qifeng Chen, Vladlen Koltun. 2017

1,008 citations recorded by the source corpus.

We present an approach to synthesizing photographic images conditioned on semantic layouts. Given a semantic label map, our approach produces an image with photographic appearance that conforms to the input layout. The approach thus functions as a rendering engine that takes a two-dimensional semantic specification of the scene and produces a corresponding photographic image. Unlike recent and contemporaneous work, our approach does not rely on adversarial training. We show that photographic images can be synthesized from semantic layouts by a single feedforward network with appropriate structure, trained end-to-end with a direct regression objective. The presented approach scales seamlessly to high resolutions; we demonstrate this by synthesizing photographic images at 2-megapixel resolution, the full resolution of our training data. Extensive perceptual experiments on datasets of outdoor and indoor scenes demonstrate that images synthesized by the presented approach are considerably more realistic than alternative approaches.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

CNN-Generated Images Are Surprisingly Easy to Spot… for Now (2019, methodology): “This includes Cascaded Reﬁnement Networks (CRN) [11], which synthesizes images in a coarse-to-ﬁne manner, and the recent Implicit Maximum Likelihood Estimation (IMLE) conditional image translation model [26].”
Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, methodology): “Human perception, which is also an important and reliable measure to evaluate synthesis quality, has been used in [71], [45], [72], [63].”
Progressive Growing of GANs for Improved Quality, Stability, and Variation (2017, neutral): “This approach has conceptual similarly to recent work by Chen & Koltun (2017).”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Unlike many GAN-based methods [12, 72, 97, 45, 98], D2C does not need to optimize an inversion procedure at evaluation time, and thus the latent value is much faster to compute; D2C is also better at retaining fine-grained features of the original image due to the reconstruction loss.”
Taming Transformers for High-Resolution Image Synthesis (2020, background): “Dataset ours SPADE [46] Pix2PixHD (+aug) [65] CRN [9]”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, methodology): “In Figure 19 and 20, we show additional synthesis re-sults from the proposed method on the ADE20K-outdoor and Cityscapes datasets with comparison to those from the CRN [7], SIMS [35], and pix2pixHD [40] methods.”
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (2018, methodology): “For example, features from the VGG architecture [51] have been used on tasks such as neural style transfer [17], image superresolution [23], and conditional image synthesis [14, 8].”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, methodology): “Recently, several works learn to produce a fixed number of discrete outputs given the same input [5, 15] or synthesize diverse modes controlled by a latent code that encodes the entire image [64].”
Multimodal Unsupervised Image-to-Image Translation (2018, methodology): “The perceptual loss, often computed as a distance in the VGG [75] feature space between the output and the reference image, has been shown to beneﬁt image-to-image translation when paired supervision is available [13, 20].”
Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis (2018, methodology): “We found that using the perceptual loss improves the stability of GAN training and the quality of object shapes, as discussed in [3, 31, 2].”
Speech Denoising with Deep Feature Losses (2018, background): “, image stylization, synthesis and reflection separation [25, 26, 27]).”
Toward Multimodal Image-to-Image Translation (2017, background): “An effective option explored by concurrent work [2, 3, 13] is to use a mixture of models.”
Connecting Vision and Language with Localized Narratives (2019, background): “Generating an image conditioned on a semantic segmentation map is a well-studied application [8, 22, 42, 56].”
Image Generation from Scene Graphs (2018, methodology): “The output image Î is generated from the layout using a cascaded refinement network (CRN) [6], shown in the right”
Playing for Benchmarks (2017, methodology): “This experimental protocol was adopted from concurrent work on photographic image synthesis [9].”
Gibson Env: Real-World Perception for Embodied Agents (2018, positive): “We found our final setup to produce superior rendering results to GAN based losses (consistent with some recent works [19]).”
Generating Multiple Objects at Spatially Distinct Locations (2019, background): “…that try to exert some control over the image layout utilize Generative Adversarial Nets (Goodfellow et al., 2014), Refinement Networks (e.g. Chen & Koltun (2017); Xu et al. (2018a)), recurrent attention-based models (e.g. Mansimov et al. (2016)), autoregressive models (e.g. Reed et al. (2016c)),…”
Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks (2018, background): “Pixel-level domain adaptation differs from image-to-image translation techniques [23, 10, 67], which deal with the easier task of learning such a re-stylization from matching pairs of examples from both domains.”
Feature Generating Networks for Zero-Shot Learning (2017, background): “Therefore, training data generation has become a hot research topic [10, 18, 11, 37, 48, 41].”
Photographic Text-to-Image Synthesis with a Hierarchically-Nested Adversarial Network (2018, background): “[3] present a cascaded refinement network to synthesize high-resolution scenes from semantic maps.”