PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph

By Yikang Li, Tao Ma, Yeqi Bai, Nan Duan, Sining Wei, Xiaogang Wang. 2019

110 citations recorded by the source corpus.

Despite some exciting progress on high-quality image generation from structured(scene graphs) or free-form(sentences) descriptions, most of them only guarantee the image-level semantical consistency, i.e. the generated image matching the semantic meaning of the description. They still lack the investigations on synthesizing the images in a more controllable way, like finely manipulating the visual appearance of every object. Therefore, to generate the images with preferred objects and rich interactions, we propose a semi-parametric method, PasteGAN, for generating the image from the scene graph and the image crops, where spatial arrangements of the objects and their pair-wise relationships are defined by the scene graph and the object appearances are determined by the given object crops. To enhance the interactions of the objects in the output, we design a Crop Refining Network and an Object-Image Fuser to embed the objects as well as their relationships into one map. Multiple losses work collaboratively to guarantee the generated images highly respecting the crops and complying with the scene graphs while maintaining excellent image quality. A crop selector is also proposed to pick the most-compatible crops from our external object tank by encoding the interactions around the objects in the scene graph if the crops are not provided. Evaluated on Visual Genome and COCO-Stuff dataset, our proposed method significantly outperforms the SOTA methods on Inception Score, Diversity Score and Frechet Inception Distance. Extensive experiments also demonstrate our method's ability to generate complex and diverse images with given objects.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, background): “In [40], an external memory bank is introduced, consisting of objects cropped from real images in training, which are retrieved and pasted in generating images from layouts at the resolution of 64×64.”
Object-Centric Image Generation from Layouts (2020, background): “Finally, there is a growing body of literature involving semi-parametric (Qi et al. 2018; Li et al. 2019c) models that use ground-truth training images to aid generation.”
Text-to-Image Generation Grounded by Fine-Grained User Attention (2020, background): “Several other approaches also use intermediate scene graph representations [18, 41] for improving image synthesis.”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, methodology): “PasteGAN [157] is a semiparametric method for image generation based on scene graphs and object cropping, designed to improve the visual”
Semantic Object Accuracy for Generative Text-to-Image Synthesis (2019, background): “[34] generate images from scene graphs and allow the model to crop objects from other images to paste them into the generated image at the correct location.”
Adversarial Text-to-Image Synthesis: A Review (2021, background): “…+ dialogue [93], [95], [99] caption + layout [104], [97], [108], [103] caption + semantic masks [109], [110], [113], [114], [115], [38] scene graphs [116], [121], [122], [124], [125] multiple captions [91], [92] multiple captions + mouse traces [126] Table 2: Methods grouped by their supervision.”
Text as Neural Operator:Image Manipulation by Text Instruction (2020, background): “Particularly, text-to-image synthesis [52, 20, 50, 58, 29, 51, 26] learns a mapping from textual descriptions to images.”
RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval (2020, methodology): “– PasteGAN [37]: The PasteGAN approach is most related to our work as it uses the pre-trained embedding function to retrieve candidate patches.”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “Traditional methods [95, 116, 154] mainly predict an image-like layout from SGs, then generate images based on the layout.”