Text2Scene: Generating Compositional Scenes From Textual Descriptions

By Fuwen Tan, Song Feng, Vicente Ordonez. 2018

101 citations recorded by the source corpus.

In this paper, we propose Text2Scene, a model that generates various forms of compositional scene representations from natural language descriptions. Unlike recent works, our method does NOT use Generative Adversarial Networks (GANs). Text2Scene instead learns to sequentially generate objects and their attributes (location, size, appearance, etc) at every time step by attending to different parts of the input text and the current status of the generated scene. We show that under minor modifications, the proposed framework can handle the generation of different forms of scene representations, including cartoon-like scenes, object layouts corresponding to real images, and synthetic images. Our method is not only competitive when compared with state-of-the-art GAN-based methods using automatic metrics and superior based on human judgments but also has the advantage of producing interpretable results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Improving Text-to-Image Synthesis Using Contrastive Learning (2021, methodology): “Recently, a great number of studies [1, 4, 6, 12, 13, 14, 17, 18, 19, 22, 23, 24, 25, 32, 33, 36, 42, 43, 44, 45, 46, 47, 48] present promising results on the text-to-image synthesis task, most of which make use of GANs as the backbone model.”
Text-to-Image Generation Grounded by Fine-Grained User Attention (2020, methodology): “In contrast with these end-to-end models, others proposed hierarchical models [15, 35, 21]; in these, object bounding boxes and segmentation masks are used as intermediate representations for realistic image generation.”
Generating Multiple Objects at Spatially Distinct Locations (2019, background): “In the future, this limitation can be avoided by extracting the relevant bounding boxes and labels directly from the image caption, as it is done for example by Hong et al. (2018b), Xu et al. (2018a), and Tan et al. (2018).”
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers (2020, background): “There is another line of works predicting object layouts from text and generating image based on the layouts (Hong et al., 2018; Tan et al., 2019).”
RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval (2020, methodology): “ods employ parametric generative models to tackle this task. The appearance of objects and relationships among objects are captured via a graph convolution network [14,23] or a text embedding network [22,32,40,42,46], then images are synthesized with the conditional generative approach. However, cur”
Dual Contrastive Loss and Attention for GANs (2021, background): “processing [46, 71, 44, 45, 77, 62, 102], image manipulation [13, 14, 70, 1, 4, 80], texture synthesis [94, 53, 59], image inpainting [34, 52, 92, 93], and text-to-image generation [68, 99, 100, 74].”
Adversarial Text-to-Image Synthesis: A Review (2021, background): “For example, users have been asked to rank images based on the relevance of text [109], to select the image which best depicts the caption [141, 108], to rate whether any one object is identiﬁable, and how well the image aligns with the text given [140], to select the more convincing image and the…”
Interactive Image Generation Using Scene Graphs (2019, methodology): “A similar approach was taken by Tan et al. (2018), where they used attention-based object and attribute decoders to infer bounding box locations of objects in the scene.”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “StackGAN++ improves upon this idea by incorporating an additional GAN to generate 128 × 128 images between the two GANs discussed above and training all GANs jointly [207].”
Benchmark for Compositional Text-to-Image Synthesis (2021, methodology): “For synthesizing images with complex scenes, some works decompose the scene into multiple objects, and use the object layout or semantic layout as a bridge to synthesize the whole image [11, 21, 10], or generate different objects sequentially [46].”
SceneFormer: Indoor Scene Generation with Transformers (2020, background): “Text2Scene [24] solves a similar task in 2 dimensions, by iteratively placing objects into an 2D image and then ensuring consistency.”