EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning

By Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z. Qureshi, Mehran Ebrahimi. 2019

788 citations recorded by the source corpus.

Over the last few years, deep learning techniques have yielded significant improvements in image inpainting. However, many of these techniques fail to reconstruct reasonable structures as they are commonly over-smoothed and/or blurry. This paper develops a new approach for image inpainting that does a better job of reproducing filled regions exhibiting fine details. We propose a two-stage adversarial model EdgeConnect that comprises of an edge generator followed by an image completion network. The edge generator hallucinates edges of the missing region (both regular and irregular) of the image, and the image completion network fills in the missing regions using hallucinated edges as a priori. We evaluate our model end-to-end over the publicly available datasets CelebA, Places2, and Paris StreetView, and show that it outperforms current state-of-the-art techniques quantitatively and qualitatively. Code and models available at: this https URL

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, background): “1 [58] Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z.”
Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “Several works suggest two-stage methods that use completion of other structure types as an intermediate step: salient edges in [32], semantic segmentation maps in [44], foreground object contours in [48], gradient maps in [52], and edge-preserved smooth images in [36].”
Regionwise Generative Adversarial Image Inpainting for Large Missing Areas (2019, methodology): “More recently, a two-stage adversarial model that consists of an edge generator followed by an image completion network was proposed in [13].”
Free-Form Image Inpainting With Gated Convolution (2018, background): “The latter approach [15, 49, 45, 46, 38, 37, 48, 26, 52, 33, 35, 19] can exploit semantics learned from large scale datasets to synthesize contents in nonstationary images in an end-to-end fashion.”
Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, background): “Inpainting (Liu et al., 2020; Nazeri et al., 2019) is the task of ﬁlling masked regions of an audio with new content since parts of the audio are corrupted or undesired.”
Coarse-to-Fine Image Inpainting via Region-wise Convolutions and Non-Local Correlation (2019, methodology): “Following [Nazeri et al., 2019], we investigate the performance of different methods using the following quantitative metrics: 1) `1 error, 2) `2 error, 3) peak signal-to-noise ratio (PSNR), and 4) structure similarity index (SSIM).”
A Survey of Human-in-the-loop for Machine Learning (2021, methodology): “The image inpainting frameworks proposed by previous studies can be bifurcated as exemplarbased [122, 123, 124] approaches and deep learning-based [125, 126, 127] methods.”
Flow-edge Guided Video Completion (2020, methodology): “We build our ﬂow edge completion network upon the publicly available oﬃcial implementation of EdgeConnect [25] 1 .”
Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations (2020, background): “The dilated convolutions [13,24] increase the size of the receptive ﬁeld to perceive encoder features.”
Image Inpainting with Learnable Feature Imputation (2020, methodology): “Several methods propose combining the input image with auxiliary information, such as user sketches [17], edges [31], or examplar-based inpainting [7].”
M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection (2021, methodology): “Perceptual Loss Perceptual loss is usually used in face inpainting approaches [39, 64] to measure the similarity between the restored”
Learning Oracle Attention for High-Fidelity Face Completion (2020, background): “Moreover, different types of structural information have been extracted by networks in order to act as prior knowledge for assisting image inpainting; for example, segmentation maps [25], object contours [28], edge [17], facial landmarks [23], and face parsing [13].”
Cali-Sketch: Stroke Calibration and Completion for High-Quality Face Image Generation from Poorly-Drawn Sketches (2019, methodology): “There are some interactive face image modiﬁcation methods under the framework of image inpainting [10], [11], [12], [13].”
Image Inpainting: A Review (2019, background): “The authors in [87] proposed a two-stage adversarial model named EdgeConnect by providing a generator for edge followed by an image inpainting model.”
Learning Joint Spatial-Temporal Transformations for Video Inpainting (2020, methodology): “e-form masks, including stationary masks and moving masks [6,16,18]. Because free-form masks are closer to real masks and have been proved to be eective for training and evaluating inpainting models [5,6,20,23]. Specically, for testing stationary masks, we generate stationary random shapes as testin”
Applications of Generative Adversarial Networks (GANs): An Updated Review (2019, background): “[83] proposed a two-stage adversarial model EdgeConnect that comprises of an edge generator followed by an image completion network.”
Region Normalization for Image Inpainting (2019, methodology): “4 Network Architecture EdgeConnect(EC) (Nazeri et al. 2019) consists of an edge generator and an image generator.”
Image Inpainting With Learnable Bidirectional Attention Maps (2019, background): “[19] proposed a two-stage model EdgeConnect first predicting salient edges and then generating inpainting result guided by edges.”