Unsupervised Attention-guided Image to Image Translation

By Youssef A. Mejjati, Christian Richardt, James Tompkin, Darren P. Cosker, Kwang In Kim. Neural Information Processing Systems, 2018

323 citations recorded by the source corpus.

Current unsupervised image-to-image translation techniques struggle to focus their attention on individual objects without altering the background or the way multiple objects interact within a scene. Motivated by the important role of attention in human perception, we tackle this limitation by introducing unsupervised attention mechanisms which are jointly adversarially trained with the generators and discriminators. We empirically demonstrate that our approach is able to attend to relevant regions in the image without requiring any additional supervision, and that by doing so it achieves more realistic mappings compared to recent approaches.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/11211bb7f7528a4d718d7abbe4981cc28eda13d2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

HoloGAN: Unsupervised Learning of 3D Representations From Natural Images (2019, background): “Most recent work on GAN architectures has focused on improving training stability or visual fidelity of the generated images, such as multi-resolution GANs [27, 61], or self-attention generators [1, 62].”
DiffEdit: Diffusion-based semantic image editing with mask guidance (2022, methodology): “…involve training an end-to-end architecture with a proxy objective before being adapted to editing at inference time, based on GANs (Li et al., 2020b;a; Ma et al., 2018; Alami Mejjati et al., 2018; Mo et al., 2018) or transformers (Wang et al., 2022a; Brown et al., 2022; Issenhuth et al., 2021).”
InstaGAN: Instance-aware Image-to-Image Translation (2018, background): “, 2017) and Attention-GAN (Mejjati et al., 2018) use segmentation masks or predicted attentions, but only to attach the background to the (translated) cropped instances.”
A Survey of Unsupervised Deep Domain Adaptation (2018, methodology): “g to the same input image). A modification to CycleGAN explored by Li et al. [136] uses separate batch normalization for each domain (an idea similar to AdaBN discussed in Section3.3). Mejjati et al. [3] and Chen et al. [38] improve results with attention, learning which areas of the images on which”
A General Survey on Attention Mechanisms in Deep Learning (2022, background): “In comparison to the total amount of research that has been done on attention models, research on attention models for semi-supervised learning [123], [124] or unsupervised learning [125], [126] has received limited attention and has only become active recently.”
Describe What to Change: A Text-guided Unsupervised Image-to-image Translation Approach (2020, background): “Attention mechanisms have proven successful in preserving the visual information that should not bemodified in an image-to-image translation [29, 30, 34, 40].”
Self-derived organ attention for unpaired CT-MRI deep domain adaptation based MRI segmentation (2020, methodology): “Furthermore, as shown in Mejjati et al (2018), computation of self-attention in an unsupervised way requires pre-training of this network for several epochs before it can be combined with the self-attention guided I2I translations.”
Image-to-Image Translation: Methods and Applications (2021, background): “Chen et al. [141] and Mejjati [12] almost simultaneously proposed attention GAN and attention- guided I2I, respectively, focusing on achieving an I2I translation of individual objects without altering the background, as shown in Fig.10 (local).”
Alleviating Semantics Distortion in Unsupervised Low-Level Image-to-Image Translation via Structure Consistency Constraint (2022, methodology): “Moreover, [5, 40, 48, 56, 57, 60, 68, 71] use the attention-based/ pretrained model or pre-define functions to preserve the semantics during translation.”
Text as Neural Operator:Image Manipulation by Text Instruction (2020, background): “The input context can be images [18, 55, 24, 16, 36], audio sequences [25], human poses [33], or semantic segmentation masks [48, 40, 28].”
U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation (2019, background): “In addition, AGGAN (Mejjati et al. (2018)) improved the performance of image translation by using attention mechanism to distinguish between foreground and background.”