Perceptual Adversarial Networks for Image-to-Image Transformation

By Chaoyue Wang, Chang Xu, Chaohui Wang, Dacheng Tao. 2017

392 citations recorded by the source corpus.

In this paper, we propose perceptual adversarial networks (PANs) for image-to-image transformations. Different from existing application driven algorithms, PAN provides a generic framework of learning to map from input images to desired images (<xref ref-type="fig" rid="fig1">Fig. 1</xref>), such as a rainy image to its de-rained counterpart, object edges to photos, and semantic labels to a scenes image. The proposed PAN consists of two feed-forward convolutional neural networks: the image transformation network <inline-formula> <tex-math notation="LaTeX">$T$ </tex-math></inline-formula> and the discriminative network <inline-formula> <tex-math notation="LaTeX">$D$ </tex-math></inline-formula>. Besides the generative adversarial loss widely used in GANs, we propose the perceptual adversarial loss, which undergoes an adversarial training process between the image transformation network <inline-formula> <tex-math notation="LaTeX">$T$ </tex-math></inline-formula> and the hidden layers of the discriminative network <inline-formula> <tex-math notation="LaTeX">$D$ </tex-math></inline-formula>. The hidden layers and the output of the discriminative network <inline-formula> <tex-math notation="LaTeX">$D$ </tex-math></inline-formula> are upgraded to constantly and automatically discover the discrepancy between the transformed image and the corresponding ground truth, while the image transformation network <inline-formula> <tex-math notation="LaTeX">$T$ </tex-math></inline-formula> is trained to minimize the discrepancy explored by the discriminative network <inline-formula> <tex-math notation="LaTeX">$D$ </tex-math></inline-formula>. Through integrating the generative adversarial loss and the perceptual adversarial loss, <inline-formula> <tex-math notation="LaTeX">$D$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$T$ </tex-math></inline-formula> can be trained alternately to solve image-to-image transformation tasks. Experiments evaluated on several image-to-image transformation tasks (e.g., image de-raining and image inpainting) demonstrate the effectiveness of the proposed PAN and its advantages over many existing works.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet (2020, background): “By dint of automatic feature engineering, deep neural networks (DNNs) have achieved remarkable success in various computer vision tasks, such as image classification [37, 36, 41, 33, 32, 15, 39], visual generation [34, 35], image retrieval [40, 7, 8, 12] and semantic comprehension [18, 17].”
Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis (2018, methodology): “We found that using the perceptual loss improves the stability of GAN training and the quality of object shapes, as discussed in [3, 31, 2].”
Image Fine-grained Inpainting (2020, positive): “he intermediate layers of well-trained VGG19 [23] model, which can be written as L fm vgg = X5 l=1 wl l I gt l I output 1 N l Igt ; (8) where N l Igt is the number of elements in l I gt . Inspired by [34], we also introduce local branch in discriminator feature matching loss L fm dis, which is reaso”
On Data Augmentation for GAN Training (2020, background): “GAN has achieved remarkable results in various tasks, for example: image synthesis [2], [3], [4], [5], image transformation [6], [7], [8], super-resolution [9], [10], text to image [11], [12], video captioning [13].”
Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models (2022, background): “adversarial networks [47]) were found successful in modeling underlying image background structures when simply applied to these problems.”
Deep Learning for Multigrade Brain Tumor Classification in Smart Healthcare Systems: A Prospective Survey (2020, background): “…techniques and are widely used for many applications in diverse domains, such as medical image synthesis [127], [128], compressive sensing MRI [129], [130], superresolution [130], security [131], classi-ﬁcation [132], image-to-image transformation [133], and many other useful purposes.”
How Generative Adversarial Networks and Its Variants Work : An Overview of GAN (2017, methodology): “In addition, we can transform an image to other specific domains [140],[122],[133] and perform various tasks such as an object transfiguration [139], super resolution [62], 3D image generation [128], [32], and control attributes of data [72],[4],[88],[36].”
Neural Compatibility Modeling With Probabilistic Knowledge Distillation (2020, background): “Deep neural networks have achieved distinguished performance in various application domains ranging from natural language processing [14][15] to computer vision [16][17][18].”
MedGAN: Medical Image Translation using GANs (2018, methodology): “For MR motion correction, pix2pix and PAN succeeded in producing globally consistent MR images, albeit with blurry details.”
A Sparse-View CT Reconstruction Method Based on Combination of DenseNet and Deconvolution (2018, background): “It has been previously demonstrated that convolution has excellent performance on image restoration from extracted feature-maps [35], [62], [63].”
How Generative Adversarial Networks and Their Variants Work (2017, background): “Their capability to represent complex and highdimensional data can be utilized in treating images [2, 12, 57, 65, 127, 133, 145], videos [122, 125,”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, methodology): “AN [100], SR-DGAN [101], T-GAN [102] Texture synthesis: MGAN [103], SGAN [105], PSGAN [106] Sketch synthesis: TGAN [107], Sketchy-GAN [110], CA-GAN [111] Image-to-image translation: Pix2pix [34], PAN [112], CycleGAN [113], Disco-GAN [114], Dual-GAN [115], StarGAN [116], UNIT [117], MUNIT [118], DRIT”
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers (2021, background): “Such deep features have been widely used in image generation [5,9,24,26,27,36,39,63] with the goal of synthesizing indistinguishable realistic images.”
Image De-Raining Using a Conditional Generative Adversarial Network (2017, methodology): “sed method [17] (TIP ’17) CCR: Convolutional-coding based method [27] (WACV ’17) DDN: Deep Detail Network method [16] (CVPR ’17) JORDER: CNN-based method [18] (CVPR ’17) PAN: GAN-based method [68] (TIP ’18) 1) Evaluation on synthetic dataset: In the ﬁrst set of experiments, we evaluate the propo”