ChatPainter: Improving Text to Image Generation using Dialogue

By Shikhar Sharma, Dendi Suhubdy, Vincent Michalski, Samira Ebrahimi Kahou, Yoshua Bengio. 2018

102 citations recorded by the source corpus.

Synthesizing realistic images from text descriptions on a dataset like Microsoft Common Objects in Context (MS COCO), where each image can contain several objects, is a challenging task. Prior work has used text captions to generate images. However, captions might not be informative enough to capture the entire image and insufficient for the model to be able to understand which objects in the images correspond to which words in the captions. We show that adding a dialogue that further describes the scene leads to significant improvement in the inception score and in the quality of generated images on the MS COCO dataset.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

Object-Centric Image Generation from Layouts (2020, background): “2019) or even multi-turn conversations (Sharma et al. 2018; El-Nouby et al. 2019; Li et al. 2019b) have gained significant interest.”
Vector Quantized Diffusion Model for Text-to-Image Synthesis (2021, background): “In the past few years, Generative Adversarial Networks (GANs) [18] have shown promising results on many tasks [19–21],, especially text-to-image generation [5, 8, 9, 14, 17, 25, 27, 31–34, 38, 43, 44, 50, 51, 58, 60–63, 67–73].”
Text-to-Image Generation Grounded by Fine-Grained User Attention (2020, background): “Such interfaces would support various applications, allowing users to rapidly create interactive scenes without specialized software [6], assist the creative process of an artist [13], or generate realistic images that are more closely grounded to intended descriptions [33].”
Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction (2018, background): “This lack of good generators may be due to the limited information content of captions, which are not rich enough to describe an entire image [2].”
Image Generation From Layout (2018, background): “Conditional image generation approaches generate images conditioned on additional input information, including entire source image [13, 23, 32, 46, 50, 51, 52], sketches [13, 36, 43, 45, 52], scene graphs [14], dialogues [16, 37] and text descriptions [25, 33, 41, 47].”
StoryGAN: A Sequential Conditional GAN for Story Visualization (2018, background): “For example, this has been explored in dialogue-to-image generation, where the input is a complete dialogue session rather than a single sentence [29].”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “Dialogue-based interactions have also been studied to control image synthesis by improving complex scene generation progressively [227]–[231].”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, background): “[156] provided a dialogue interface that uses textual information from the dialogue to obtain more detailed information about the image.”
Adversarial Text-to-Image Synthesis: A Review (2021, background): “Motivated by the fact that a single sentence might not be informative enough to describe a scene containing several interacting objects, Sharma et al. proposed ChatPainter [93] to leverage dialog data.”
Text2Scene: Generating Compositional Scenes From Textual Descriptions (2018, methodology): “g or mapping images to text [5, 18, 13, 10, 27, 28, 19, 2]. However, there has been less work in the opposite direction of mapping text to images, this is, using text as an input to synthesize images [9, 22, 30, 23]. Most of these recent approaches have leveraged conditional Generative Adversarial N”
Semantic Object Accuracy for Generative Text-to-Image Synthesis (2019, background): “As a result, many approaches now use attention mechanisms to attend to speciﬁc words of the sentence [7], use intermediate representations such as scene layouts [2], condition on additional information such as object bounding boxes [3] or perform interactive image reﬁnement [12].”
RiFeGAN: Rich Feature Generation for Text-to-Image Synthesis From Prior Knowledge (2020, background): “Generating realistic images from text descriptions is one of the most active research areas in recent years [10, 12, 16, 21, 24, 26, 33, 37].”
Layout2image: Image Generation from Layout (2020, background): “2018), dialogues (Kim et al. 2017; Sharma et al. 2018) and text descriptions (Mansimov et al.”
ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language (2019, background): “, image captioning [26, 59, 64, 38], image-text matching [15, 30, 34, 24, 16], and text-to-image generation [53, 16, 57]), but there is little work on vision and language in 3D.”
On the Evaluation of Conditional GANs (2019, background): “More recently, dialogue-conditioned image generation has also been introduced [8, 36].”