Semi-Parametric Image Synthesis

By Xiaojuan Qi, Qifeng Chen, Jiaya Jia, Vladlen Koltun. 2018

177 citations recorded by the source corpus.

We present a semi-parametric approach to photographic image synthesis from semantic layouts. The approach combines the complementary strengths of parametric and nonparametric techniques. The nonparametric component is a memory bank of image segments constructed from a training set of images. Given a novel semantic layout at test time, the memory bank is used to retrieve photographic references that are provided as source material to a deep network. The synthesis is performed by a deep network that draws on the provided photographic material. Experiments on multiple semantic segmentation datasets show that the presented approach yields considerably more realistic images than recent purely parametric techniques.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Object-Centric Image Generation from Layouts (2020, background): “r this task, it is common to introduce unsupervised mask generators [53,35] to provide estimated shapes for this conditioning. Finally, there is a growing body of literature involving semi-parametric [43,28] models that use ground-truth training image to aid generation. We consider the case of such ”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, methodology): “Recent work has achieved photorealistic semantic image synthesis results [43, 47] on the Cityscapes dataset.”
Instance-Conditioned GAN (2021, methodology): “Existing methods for complex scene generation, where natural looking scenes contain multiple objects, most often aim at controllability and rely on detailed conditionings such as a scene graphs [3, 25], bounding box layouts [48–50, 58], semantic segmentation masks [9, 43, 44, 52, 55] or more recentl”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, background): “Most sensor fusion works consider perception tasks, e.g. object detection [14], [15], [16], [18], [19], [20], [21], [22], [23], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60] and motion forecasting [24], [25], [26], [27], [28], [29], [30], [49], [61], [62].”
SEAN: Image Synthesis With Semantic Region-Adaptive Normalization (2019, unknown): “While multiple very good frameworks exist to tackle this problem [23, 8, 40, 47], the currently best architecture is SPADE [39] (also called GauGAN).”
Focal Frequency Loss for Image Reconstruction and Synthesis (2020, background): “SIMS [45] obtains the best FID but poor segmentation scores on Cityscapes in that it directly stitches the training image patches from a memory bank while not keeping the exactly consistent positions.”
Enhancing Photorealism Enhancement (2021, background): “graphic images [10], [11], [12], [13], [14], [15], [16], and to attempt to bridge the appearance gap between synthetic”
Mask-Guided Portrait Editing With Conditional GANs (2019, background): “[37] which combines neural networks to improve quality.”
You Only Need Adversarial Supervision for Semantic Image Synthesis (2020, methodology): “Besides GANs, Chen & Koltun (2017) proposed to use a cascaded refinement network (CRN) for high-resolution semantic image synthesis, and SIMS (Qi et al., 2018) extended it with a non-parametric component, serving as a memory bank of source material to assist the synthesis.”
Manipulating Attributes of Natural Scenes via Hallucination (2018, methodology): “Alternatively, some efforts on image-to-image translation has been made to increase the realism and resolution with multi-scale approaches [Chen and Koltun 2017; Park et al. 2019; Qi et al. 2018; Wang et al. 2018].”
Text2Scene: Generating Compositional Scenes From Textual Descriptions (2018, methodology): “As the composite images may exhibit gaps between patches, we also leverage the stitching network in [24] for post-processing.”
Cross-Domain Correspondence Learning for Exemplar-Based Image Translation (2020, methodology): “Non-parametric or semi-parametric approaches [39, 2] synthesize images by compositing the image fragments retrieved from a large database.”
Wide-Context Semantic Image Extrapolation (2019, methodology): “As shown in recent work [32, 19], this line of methods use retrieved object segments matched by input to fill the unknown region in advance, and regress raw material.”
RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval (2020, methodology): “Another stream of work has recently emerged to explore the semi-parametric model that leverages a memory bank to retrieve the objects for synthesizing the image [28,41].”
PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph (2019, methodology): “Such information could be of different forms, like image labels [10, 11, 12], textual descriptions [1, 2, 13], scene graphs [4] and semantic segmentations [14, 3].”
Cali-Sketch: Stroke Calibration and Completion for High-Quality Face Image Generation from Poorly-Drawn Sketches (2019, methodology): “Label-based image synthesis methods [5], [16], [17] synthesize image semantically from abstract label maps, such as sparse landmarks or pixel-wise segmentation maps.”
Multi-Modal Fusion Transformer for End-to-End Autonomous Driving (2021, background): “object detection [22, 12, 66, 7, 44, 31, 34, 61, 33, 37] and motion forecasting [36, 5, 35, 63, 6, 19, 38].”
Improving Augmentation and Evaluation Schemes for Semantic Image Synthesis (2020, methodology): “Besides GANs, CRN [8] and SIMS [42] train a cascaded refinement convolutional network with a regression loss.”
Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis (2019, methodology): “SIMS [26] developed a semi-parametric approach, by retrieving fragments and refining them with a refinement network.”