Semantic Image Inpainting with Progressive Generative Networks

By Haoran Zhang, Zhenzhen Hu, Wangmeng Zuo, M. Wang. 2018

146 citations recorded by the source corpus.

Recently, image inpainting task has revived with the help of deep learning techniques. Deep neural networks, especially the generative adversarial networks~(GANs) make it possible to recover the missing details in images. Due to the lack of sufficient context information, most existing methods fail to get satisfactory inpainting results. This work investigates a more challenging problem, e.g., the newly-emerging semantic image inpainting - a task to fill in large holes in natural images. In this paper, we propose an end-to-end framework named progressive generative networks~(PGN), which regards the semantic image inpainting task as a curriculum learning problem. Specifically, we divide the hole filling process into several different phases and each phase aims to finish a course of the entire curriculum. After that, an LSTM framework is used to string all the phases together. By introducing this learning strategy, our approach is able to progressively shrink the large corrupted regions in natural images and yields promising inpainting results. Moreover, the proposed approach is quite fast to evaluate as the entire hole filling is performed in a single forward pass. Extensive experiments on Paris Street View and ImageNet dataset clearly demonstrate the superiority of our approach. Code for our models is available at https://github.com/crashmoon/Progressive-Generative-Networks.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, neutral): “Another trend is progressive approaches [62, 12, 25, 61].”
Image Inpainting with Learnable Feature Imputation (2020, methodology): “ent scales [16,25], speciﬁc modules to connect spatially distant features [39,44,45,47], patch-based discriminators [48,49], multi-column generators [40], or progressively inpainting the missing area [11,50]. In contrast to these methods, we ensure consistency over multiple resolutions by connecting”
UCTGAN: Diverse Image Inpainting Based on Unsupervised Cross-Space Translation (2020, background): “These approaches, such as [28, 8, 15, 20, 24], often generate distorted structures and blurry textures inconsistent with the visible regions.”
Recurrent Feature Reasoning for Image Inpainting (2020, methodology): “An alternative scheme is to inpaint in a progressive fashion from the hole boundary to the center [31, 9].”
Image Inpainting With Learnable Bidirectional Attention Maps (2019, background): “[37] presented a progressive generative networks (PGN) for filling in holes with multiple phases, while LSTM is deployed to exploit the dependencies across phases.”
Learning Oracle Attention for High-Fidelity Face Completion (2020, background): “Recently, convolutional neural network (CNN) based methods have become the mainstream approach to image inpainting [30, 29, 12, 26, 36].”
Progressive Image Inpainting with Full-Resolution Residual Network (2019, methodology): “Further, GAN-based works [12, 21, 29, 31–33, 35] can generate visually plausible results.”
High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling (2020, background): “Zhang et al. [37] adopt a UNet generator with an LSTM in the bottleneck.”
MAT: Mask-Aware Transformer for Large Hole Image Inpainting (2022, background): “To allow for high-resolution image inpainting, a few attempts have been made to develop progressively generation systems, such as [18, 32, 41, 71, 72].”