StoryGAN: A Sequential Conditional GAN for Story Visualization

By Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Edwin Carlson, Jianfeng Gao. 2018

299 citations recorded by the source corpus.

In this work, we propose a new task called Story Visualization. Given a multi-sentence paragraph, the story is visualized by generating a sequence of images, one for each sentence. In contrast to video generation, story visualization focuses less on the continuity in generated images (frames), but more on the global consistency across dynamic scenes and characters -- a challenge that has not been addressed by any single-image or video generation methods. Therefore, we propose a new story-to-image-sequence generation model, StoryGAN, based on the sequential conditional GAN framework. Our model is unique in that it consists of a deep Context Encoder that dynamically tracks the story flow, and two discriminators at the story and image levels, to enhance the image quality and the consistency of the generated sequences. To evaluate the model, we modified existing datasets to create the CLEVR-SV and Pororo-SV datasets. Empirically, StoryGAN outperformed state-of-the-art models in image quality, contextual consistency metrics, and human evaluation.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Object-Centric Image Generation from Layouts (2020, background): “2019) or even multi-turn conversations (Sharma et al. 2018; El-Nouby et al. 2019; Li et al. 2019b) have gained significant interest.”
KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis (2020, background): “If the given database only contains images with single sentence description (which is common in some practical tasks such as Story Visualization [38] and Text-to-Video [37]), SD-GAN can not be trained.”
RiFeGAN: Rich Feature Generation for Text-to-Image Synthesis From Prior Knowledge (2020, background): “Generating realistic images from text descriptions is one of the most active research areas in recent years [10, 12, 16, 21, 24, 26, 33, 37].”
Semantic Object Accuracy for Generative Text-to-Image Synthesis (2019, background): “[38] generate images iteratively from consecutive textual commands, [39] provide interactive image editing, based on a current image and instructions on how to update the image, and [40] generate individual images for a sequence of sentences.”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, background): “[134] proposed StoryGAN, a story-image sequence generation model that sequentially visualizes stories by generating one image sequence for each sentence.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “Sequential Image Generation approach StoryGAN (Li et al., 2018a), based on the sequential conditional GAN concentrate on story by generating a sequence of images, when given a multi-sentence paragraph.”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, unknown): “consistency among generated images is required [232]–[234].”
Text as Neural Operator:Image Manipulation by Text Instruction (2020, background): “, providing the ground-truth manipulated images for the inputs of a text instruction and a reference image), existing works [8, 30] were only able to test on synthetic images.”
Describe What to Change: A Text-guided Unsupervised Image-to-image Translation Approach (2020, background): “proposed StoryGAN [28], which generates a series of images that are contextually coherent with previously generated images and with the sequence of text descriptions provided by the user.”