Prior Guided GAN Based Semantic Inpainting

By Avisek Lahiri, Arnav Kumar Jain, Sanskar Agrawal, Pabitra Mitra, Prabir Kumar Biswas. 2020

101 citations recorded by the source corpus.

Contemporary deep learning based semantic inpainting can be approached from two directions. First, and the more explored, approach is to train an offline deep regression network over the masked pixels with an additional refinement by adversarial training. This approach requires a single feed-forward pass for inpainting at inference. Another promising, yet unexplored approach is to first train a generative model to map a latent prior distribution to natural image manifold and during inference time search for the best-matching prior to reconstruct the signal. The primary aversion towards the latter genre is due to its inference time iterative optimization and difficulty to scale to higher resolution. In this paper, going against the general trend, we focus on the second paradigm of inpainting and address both of its mentioned problems. Most importantly, we learn a data driven parametric network to directly predict a matching prior for a given masked image. This converts an iterative paradigm to a single feed forward inference pipeline with around 800X speedup. We also regularize our network with structural prior (computed from the masked image itself) which helps in better preservation of pose and size of the object to be inpainted. Moreover, to extend our model for sequence reconstruction, we propose a recurrent net based grouped latent prior learning. Finally, we leverage recent advancements in high resolution GAN training to scale our inpainting network to 256X256. Experiments (spanning across resolutions from 64X64 to 256X256) conducted on SVHN, Standford Cars, CelebA, CelebA-HQ and ImageNet image datasets, and FaceForensics video datasets reveal that we consistently improve upon contemporary benchmarks from both schools of approaches.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Eric I-Chao Chang
2. Yao-Kai Liu
3. Yilun Sheng
4. Xiaoming Liu
5. Yue Dong
6. Xiao Liang
7. Jun Chen
8. Jianwei Cui

In Generative Models

1. Eric I-Chao Chang
2. Yilun Sheng
3. Xiaoming Liu
4. Yue Dong
5. Xiao Liang
6. Jianwei Cui
7. Xiaohong Liu
8. Yan Xu

Citations in the corpus

• Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “…et al., 2018; Xie et al., 2019; Xiong et al., 2019; Yan et al., 2018; Yang et al., 2017; 2019a; Yu et al., 2018; Yu et al., 2019; Zeng et al., 2019; Lahiri et al., 2020; Zhao et al., 2020a; Li et al., 2020; Zhou et al., 2020), among which (Liu et al., 2018; Yu et al., 2019) introduce partial…”
• PSCC-Net: Progressive Spatio-Channel Correlation Network for Image Manipulation Detection and Localization (2021, background): “Recent advances on image manipulation techniques [13,32,33,37] enable easy editing of raw images, such as removing unwanted objects [31, 34, 35, 67], face swapping [32], attribute changing [50], etc.”