Eye In-painting with Exemplar Generative Adversarial Networks

By Brian Dolhansky, Cristian Canton-Ferrer. 2017

117 citations recorded by the source corpus.

This paper introduces a novel approach to in-painting where the identity of the object to remove or change is preserved and accounted for at inference time: Exemplar GANs (ExGANs). ExGANs are a type of conditional GAN that utilize exemplar information to produce high-quality, personalized in-painting results. We propose using exemplar information in the form of a reference image of the region to in-paint, or a perceptual code describing that object. Unlike previous conditional GAN formulations, this extra information can be inserted at multiple points within the adversarial network, thus increasing its descriptive power. We show that ExGANs can produce photo-realistic personalized in-painting results that are both perceptually and semantically plausible by applying them to the task of closed-to-open eye in-painting in natural pictures. A new benchmark dataset is also introduced for the task of eye inpainting for future comparisons.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

1. Håkon Hukkelås
2. Dacheng Tao
3. Jie Gui
4. Jieping Ye
5. Lu Yuan
6. Paolo Favaro
7. Yonggang Wen
8. Frank Lindseth

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Xiaogang Wang
4. Ming-Hsuan Yang
5. Yi Yang
6. Xiaoou Tang
7. Lu Yuan
8. Shuicheng Yan

Citations in the corpus

• EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning (2019, background): “cent works [55, 54, 9] have shown that metrics based on deep features are closer to those based on human perception.”
• Mask-Guided Portrait Editing With Conditional GANs (2019, background): “, eyes [13, 41]) or editing attributes of faces [35, 42].”
• Image Inpainting with Learnable Feature Imputation (2020, methodology): “For image inpainting, the adversarial loss is often combined with other objectives to improve training stability, such as pixel-wise reconstruction [7,16,25,33], perceptual loss [39,51], semantic loss [25], or style loss [41].”
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, methodology): “GANs have also been used in other face-related applications, such as facial attribute changes [222] and portrait editing [223], [224], [225], [226].”
• A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, background): “Exemplar GAN (Ex-GAN) [44] uses the exemplar image information as a reference image for eye-closing to eye-opening tasks and produces a high-quality result.”
• Auto-Embedding Generative Adversarial Networks For High Resolution Image Synthesis (2019, background): “Moreover, GANs have also been applied to range of other interesting applications, such as text to image synthesis [25], [33], [39], [56], super-resolution [16], [46], image inpainting [5], [49], [54] and so on.”
• Coherent Semantic Attention for Image Inpainting (2019, unknown): “Learning approaches [38, 22, 31, 37, 41, 7, 42] often use deep learning and GAN strategy to generate pixels of the hole.”
• EdgeConnect: Structure Guided Image Inpainting using Edge Prediction (2019, background): “Dolhansky and Ferrer [10] demonstrate the importance of exemplar information for inpainting.”
• Faceshop (2018, methodology): “In addition, these results underline the flexibility of our system, which is able to realize a variety of intended user edits without being restricted to specific operations (like opening eyes [Dolhansky and Ferrer 2017]).”