Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction

By Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, Graham W.Taylor. 2018

129 citations recorded by the source corpus.

Conditional text-to-image generation is an active area of research, with many possible applications. Existing research has primarily focused on generating a single image from available conditioning information in one step. One practical extension beyond one-step generation is a system that generates an image iteratively, conditioned on ongoing linguistic input or feedback. This is significantly more challenging than one-step generation tasks, as such a system must understand the contents of its generated images with respect to the feedback history, the current feedback, as well as the interactions among concepts present in the feedback history. In this work, we present a recurrent image generation model which takes into account both the generated output up to the current step as well as all past instructions for generation. We show that our model is able to generate the background, add new objects, and apply simple transformations to existing objects. We believe our approach is an important step toward interactive generation. Code and data is available at: https://www.microsoft.com/en-us/research/project/generative-neural-visual-artist-geneva/.

Subfields: Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

In Multimodal Learning

Citations in the corpus

Object-Centric Image Generation from Layouts (2020, background): “2019) or even multi-turn conversations (Sharma et al. 2018; El-Nouby et al. 2019; Li et al. 2019b) have gained significant interest.”
Vector Quantized Diffusion Model for Text-to-Image Synthesis (2021, background): “In the past few years, Generative Adversarial Networks (GANs) [18] have shown promising results on many tasks [19–21],, especially text-to-image generation [5, 8, 9, 14, 17, 25, 27, 31–34, 38, 43, 44, 50, 51, 58, 60–63, 67–73].”
Improving Text-to-Image Synthesis Using Contrastive Learning (2021, methodology): “Most of the recent works [45, 46, 42, 13, 23, 48, 43, 18, 22, 17, 1, 12, 6, 19, 33, 4] on the text-to-image synthesis problem are based on GANs.”
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, background): “[89]), until further research has examined questions of efficacy and utility, since text and image convey meaning in distinct ways and with distinct limitations.”
Text-to-Image Generation Grounded by Fine-Grained User Attention (2020, background): “Others have explored dialogue-based interactions in which users incrementally provide instructions to refine and adjust generated scenes [33, 8].”
MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing (2023, background): “Instruction-guided editing, as initially proposed in various studies [11, 13, 42], enables users to edit images without requiring elaborate descriptions or region masking.”
Adversarial Text-to-Image Synthesis: A Review (2021, methodology): “Another approach taken by [38] is to use instance masks for the desired objects and split the image generation process into foreground (objects), and background synthesis before blending them into the ﬁnal image (similar to [160] and iterative generation as in [161, 162, 93]).”
On the Evaluation of Conditional GANs (2019, background): “More recently, dialogue-conditioned image generation has also been introduced [8, 36].”
ManiGAN: Text-Guided Image Manipulation (2019, background): “There are few studies focusing on image manipulation using natural language descriptions [2, 4, 6, 8, 23].”
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis (2022, unknown): “…There are mainly three types of models for text-to-image synthesis: GANbased (Tao et al., 2022; Zhu et al., 2019; Li et al., 2019; Fu et al., 2020; El-Nouby et al., 2019), autoregressive (Gu et al., 2022a; Lee et al., 2022; Ding et al., 2022) and diffusion models (Liu et al., 2021b; Nichol et…”
Sound-Guided Semantic Image Manipulation (2021, methodology): “Text-based image manipulation methods are proposed to edit the image conditionally [12, 21, 26, 30, 51].”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, methodology): “Data science: GANs have been used in data generating [214], [418]–[426], neural networks generating [427], data augmentation [428], [429], spatial representation learning [430], network embedding [431], heterogeneous information networks [432], and mobile user profiling [433].”
Describe What to Change: A Text-guided Unsupervised Image-to-image Translation Approach (2020, background): “[11, 12] rely on annotated datasets to interact with the user and to generate semantically-consistent images from textual instructions.”
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models (2023, methodology): “Evaluation Metrics To evaluate generated layouts, we report precision, recall, and accuracy based on generated bounding box counts and spatial positions [9, 17].”
History for Visual Dialog: Do we really need it? (2020, background): “conversational image editing, which is one of the target applications of visual dialog (Kim et al., 2017; Manuvinakurike et al., 2018a,b; Lin et al., 2018; El-Nouby et al., 2018).”
Semantic Object Accuracy for Generative Text-to-Image Synthesis (2019, background): “[38] generate images iteratively from consecutive textual commands, [39] provide interactive image editing, based on a current image and instructions on how to update the image, and [40] generate individual images for a sequence of sentences.”
Text as Neural Operator:Image Manipulation by Text Instruction (2020, methodology): “3 GeNeVA [8] 46.”