Recurrent Topic-Transition GAN for Visual Paragraph Generation

By Xiaodan Liang, Zhiting Hu, H. Zhang, Chuang Gan, Eric P. Xing. 2017

219 citations recorded by the source corpus.

A natural image usually conveys rich semantic content and can be viewed from different angles. Existing image description methods are largely restricted by small sets of biased visual paragraph annotations, and fail to cover rich underlying semantics. In this paper, we investigate a semi-supervised paragraph generative framework that is able to synthesize diverse and semantically coherent paragraph descriptions by reasoning over local semantic regions and exploiting linguistic knowledge. The proposed Recurrent Topic-Transition Generative Adversarial Network (RTT-GAN) builds an adversarial framework between a structured paragraph generator and multi-level paragraph discriminators. The paragraph generator generates sentences recurrently by incorporating region-based visual and language attention mechanisms at each step. The quality of generated paragraph sentences is assessed by multi-level adversarial discriminators from two aspects, namely, plausibility at sentence level and topic-transition coherence at paragraph level. The joint adversarial training of RTT-GAN drives the model to generate realistic paragraphs with smooth logical transition between sentence topics. Extensive quantitative experiments on image and video paragraph datasets demonstrate the effectiveness of our RTT-GAN in both supervised and semi-supervised settings. Qualitative results on telling diverse stories for an image verify the interpretability of RTT-GAN.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

NetGAN: Generating Graphs via Random Walks (2018, methodology): “ing techniques to enable backpropagation through sampling discrete random variables (Yu et al., 2017; Kusner & Hernández-Lobato, 2016; Li et al., 2017; Liang et al., 2017).”
Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations (2018, background): “Generative adversarial networks (GANs) have achieved remarkable success within short time for diverse tasks of generating synthetic data, such as images [1, 2, 3, 4], texts [5, 6, 7, 8], and even music [5, 9, 10, 11].”
Scenic: a language for scenario specification and scene generation (2018, background): “In particular, Generative Adversarial Networks (GANs) [17], a particular kind of neural network able to generate synthetic data, have been used to augment training sets [28, 30].”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “Related to this variant, an important line of works [66], [197], [198], [199], [200], [201] focuses on the generation of textual paragraphs that densely describe the visual content as a coherent story.”
Shaping the Narrative Arc: An Information-Theoretic Approach to Collaborative Dialogue (2019, methodology): “Previous work has used a topic-transition generative adversarial network to enforce smoothness of transition of subsequent utterances (Liang et al., 2017).”
Generative Semantic Manipulation with Contrasting GAN (2017, background): “Impressive progresses have been achieved on a wide variety of image generation [19, 24, 25], image editing [32], text generation [14] and conditional image generation such as text2image [23], image inpainting [20], and image translation [9] tasks.”
Deep Multimodal Representation Learning: A Survey (2019, background): “Most recently, generative adversarial learning strategy is further extended to multimodal cases such as text-to-image synthesis [15], [44], visual captioning [40], [43], cross-modal retrieval [30], multimodal features fusion [4], and multimodal storytelling [133].”
Language Generation with Recurrent Generative Adversarial Networks without Pre-training (2017, methodology): “Consequently, past work on using GANs for text generation has been based on pre-training (Yu et al., 2016; Li et al., 2017; Yang et al., 2017; Wu et al., 2017; Liang et al., 2017; Zhang et al., 2016; Shetty et al., 2017)”
3D Human Pose Estimation in the Wild by Adversarial Learning (2018, background): “Adversarial learning has been proven effective not only for generative tasks [16, 33, 46, 59, 47, 10, 18, 55, 21, 20, 45, 23], but also for discriminative tasks [48, 50, 7, 6, 36].”
CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving (2018, methodology): “Deep-RL or RL algorithm has been applied to a wide variety of tasks, such as object recognition [21,22,23,24,25], computer games [26], robot locomotion [27], scene navigation [28] and autonomous driving in the simulators [29,6,30].”
Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports (2019, background): “Some other efforts have been made for building variants of the hierarchical Long-Short-Term-Memory (LSTM) network (Hochreiter and Schmidhuber, 1997) to generate paragraphs (Krause et al., 2017; Yu et al., 2016; Liang et al., 2017).”
Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation (2021, background): “Most existing methods, like [15, 42, 44] follow the standard image captioning approaches and employ the encoder-decoder framework, e.g., CNN-HRNN [15, 23].”