DA-GAN: Instance-Level Image Translation by Deep Attention Generative Adversarial Networks

By Shuang Ma, Jianlong Fu, Chang Wen Chen, Tao Mei. 2018

160 citations recorded by the source corpus.

Unsupervised image translation, which aims in translating two independent sets of images, is challenging in discovering the correct correspondences without paired data. Existing works build upon Generative Adversarial Networks (GANs) such that the distribution of the translated images are indistinguishable from the distribution of the target set. However, such set-level constraints cannot learn the instance-level correspondences (e.g. aligned semantic parts in object transfiguration task). This limitation often results in false positives (e.g. geometric or semantic artifacts), and further leads to mode collapse problem. To address the above issues, we propose a novel framework for instance-level image translation by Deep Attention GAN (DA-GAN). Such a design enables DA-GAN to decompose the task of translating samples from two sets into translating instances in a highly-structured latent space. Specifically, we jointly learn a deep attention encoder, and the instance-level correspondences could be consequently discovered through attending on the learned instances. Therefore, the constraints could be exploited on both set-level and instance-level. Comparisons against several state-of-the-arts demonstrate the superiority of our approach, and the broad application capability, e.g, pose morphing, data augmentation, etc., pushes the margin of domain translation problem.1

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Imperfect ImaGANation: Implications of GANs Exacerbating Biases on Facial Data Augmentation and Snapchat Selfie Lenses (2020, background): “Unfortunately, it is known that even these conditional GAN variants are not immune to mode collapse (Ma et al., 2018).”
DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis (2019, background): “DA-GAN [12] translates each word into a sub-region of an image.”
Object-Driven Text-To-Image Synthesis via Adversarial Training (2019, methodology): “Although images with realistic texture have been synthesized on simple datasets, such as birds [29, 16] and ﬂowers [33], most existing approaches do not speciﬁcally model objects and their relations in images and thus have difﬁculties in generating complex scenes such as those in the COCO dataset…”
Unsupervised Attention-guided Image to Image Translation (2018, background): “[18], who attempt to decouple local textures from holistic shapes by attending to local objects of interest (e.”
DiffEdit: Diffusion-based semantic image editing with mask guidance (2022, methodology): “…involve training an end-to-end architecture with a proxy objective before being adapted to editing at inference time, based on GANs (Li et al., 2020b;a; Ma et al., 2018; Alami Mejjati et al., 2018; Mo et al., 2018) or transformers (Wang et al., 2022a; Brown et al., 2022; Issenhuth et al., 2021).”
Unpaired Image-to-Speech Synthesis With Multimodal Information Bottleneck (2019, background): “Since then, numerous methods have tackled the problem from an unsupervised learning perspective, eliminating the need for paired data [58, 41, 26, 5, 21].”
Improving Shape Deformation in Unsupervised Image-to-Image Translation (2018, background): “[30] examine adapting CycleGAN to wider variety in the domains—so-called instance-level translation.”
Generative Adversarial Networks: A Survey and Taxonomy (2019, background): “These properties have allowed GANs to enjoy success especially in the computer vision ﬁeld e.g., plausible image generation [29]–[33], image to image translation [2], [34]–[40], image super-resolution [26], [41]–[44] and image completion [45]–[49].”
Application of deep learning algorithms in geotechnical engineering: a short critical review (2021, background): “…et al. 2018; Karras et al. 2017, 2019; Salimans et al. 2016), face synthesis (Bao et al. 2018), image style transfer (Chen et al. 2018a; Choi et al. 2018; Ma et al. 2018b; Taigman et al. 2016), image segmentation (Zhang et al. 2018c), facial expression recognition (Peng and Wang 2018) and so on.”
Exemplar Guided Unsupervised Image-to-Image Translation (2018, background): “1 Introduction Image-to-image translation has received significant attention, as it enables numerous applications, such as image editing [34, 34], attribute/style transfer [3, 29] and domain adaptation [12, 26].”
Learning Texture Transformer Network for Image Super-Resolution (2020, methodology): “Such a design enables a joint feature embedding of LR and Ref images which creates a solid foundation for applying attention mechanism [19, 34, 31] in SR tasks.”
Learning Joint Spatial-Temporal Transformations for Video Inpainting (2020, methodology): “To eﬀectively model long-range correspondences, recent models have adopted attention modules and show promising results in image and video synthesis [21,34,36].”
Self-Supervised CycleGAN for Object-Preserving Image-to-Image Domain Adaptation (2020, background): “Witnessing the success of cycle-consistency-based approaches [36,13,32], an increasing number of researchers [4,6,22,25] made their eﬀort to the area of un-paired I2I translation.”
Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss (2019, background): “Attention mechanism is an emerging topic in natural language tasks [20] and image/video generation task [26, 37, 22, 36].”