Perceptual Pyramid Adversarial Networks for Text-to-Image Synthesis

By Lianli Gao, Daiyuan Chen, Jingkuan Song, Xing Xu, Dongxiang Zhang, Heng Tao Shen. 2019

46 citations recorded by the source corpus.

Generating photo-realistic images conditioned on semantic text descriptions is a challenging task in computer vision field. Due to the nature of hierarchical representations learned in CNN, it is intuitive to utilize richer convolutional features to improve text-to-image synthesis. In this paper, we propose Perceptual Pyramid Adversarial Network (PPAN) to directly synthesize multi-scale images conditioned on texts in an adversarial way. Specifically, we design one pyramid generator and three independent discriminators to synthesize and regularize multi-scale photo-realistic images in one feed-forward process. At each pyramid level, our method takes coarse-resolution features as input, synthesizes highresolution images, and uses convolutions for up-sampling to finer level. Furthermore, the generator adopts the perceptual loss to enforce semantic similarity between the synthesized image and the ground truth, while a multi-purpose discriminator encourages semantic consistency, image fidelity and class invariance. Experimental results show that our PPAN sets new records for text-to-image synthesis on two benchmark datasets: CUB (i.e., 4.38 Inception Score and .290 Visual-semantic Similarity) and Oxford-102 (i.e., 3.52 Inception Score and .297 Visual-semantic Similarity).

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

1. Trevor Darrell
2. Samaneh Azadi

In Computer Vision and Pattern Recognition

1. Trevor Darrell
2. Dacheng Tao
3. Anna Rohrbach
4. Dietrich Klakow
5. Luc Van Gool
6. Aditya Mogadala
7. Marimuthu Kalimuthu
8. Andrew Zisserman

In Generative Models

1. Andreas R. Dengel
2. Marimuthu Kalimuthu
3. Trevor Darrell
4. Tobias Hinz
5. Samaneh Azadi
6. Xihui Liu
7. Dietrich Klakow
8. Aditya Mogadala

In Multimodal Learning

1. Trevor Darrell
2. Anna Rohrbach
3. Dietrich Klakow
4. Marimuthu Kalimuthu
5. Aditya Mogadala
6. Dong Huk Park
7. Andreas R. Dengel
8. Tobias Hinz

Citations in the corpus

• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Further, for getting better high resolution images, coarse-resolution features are taken as input and perceptual pyramid adversarial network (PPAN) is introduced to directly synthesize multi-scale images conditioned on texts in an adversarial way (Gao et al., 2019).”
• Benchmark for Compositional Text-to-Image Synthesis (2021, background): “StackGAN [60] and several others [61, 62, 6, 15] improved the quality of synthesized images by using stacked generator”
• Adversarial Text-to-Image Synthesis: A Review (2021, background): “…there are multiple aspects that would resemble a good image (e.g., visual Input Method caption [16], [33], [40], [42], [41], [48], [35], [54], [43], [55], [58], [61], [65], [67], [68], [69], [70], [75], [80], [86], [34], [87], [128] caption + dialogue [93], [95], [99] caption + layout [104],…”