GAN Prior Embedded Network for Blind Face Restoration in the Wild

By Tao Yang, Peiran Ren, Xuansong Xie, Lei Zhang. 2021

375 citations recorded by the source corpus.

Blind face restoration (BFR) from severely degraded face images in the wild is a very challenging problem. Due to the high illness of the problem and the complex unknown degradation, directly training a deep neural network (DNN) usually cannot lead to acceptable results. Existing generative adversarial network (GAN) based methods can produce better results but tend to generate over-smoothed restorations. In this work, we propose a new method by first learning a GAN for high-quality face image generation and embedding it into a U-shaped DNN as a prior decoder, then fine-tuning the GAN prior embedded DNN with a set of synthesized low-quality face images. The GAN blocks are designed to ensure that the latent code and noise input to the GAN can be respectively generated from the deep and shallow features of the DNN, controlling the global face structure, local face details and background of the reconstructed image. The proposed GAN prior embedded network (GPEN) is easy-to-implement, and it can generate visually photo-realistic results. Our experiments demonstrated that the proposed GPEN achieves significantly superior results to state-of-the-art BFR methods both quantitatively and qualitatively, especially for the restoration of severely degraded face images in the wild. The source code and models can be found at https://github.com/yangxy/GPEN.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

StyleGAN-NADA (2021, background): “These range from image enhancement [18,50], editing [13,36] and recently even discriminative tasks such as classification and regression [27, 48].”
Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild (2024, methodology): “Various approaches employ generative priors, including GAN inversion [2, 4, 27, 55, 60], GAN encoders [9, 100], or using GAN as the core module for IR [77, 84].”
Pretraining is All You Need for Image-to-Image Translation (2022, background): “There are prior attempts [37, 9, 56, 49, 51, 33, 31] to leverage a pretrained model as generative prior for conditional image synthesis, image editing and restoration.”
DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior (2023, methodology): “DiffBIR is compared with state-of-the-art 1) BSR methods: FeMaSR [5], DASR [30], Real-ESRGAN+ [56], BSRGAN [73], SwinIR-GAN [29], StableSR [52] and PASD [66], 2) BFR methods: CodeFormer [77], DifFace [67], DMDNet [28], DR2 [61], GCFSR [18], GFP-GAN [55], GPEN [65], RestoreFormer++ [60], VQFR [16] an”
Exploiting Diffusion Prior for Real-World Image Super-Resolution (2023, neutral): “While effective, the implicit priors used in these approaches are often tailored for speciﬁc scenarios, such as limited categories [6,20,46,64] and faces [41,62,74], and therefore lack generalizability for complex real-world SR tasks.”
Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization (2023, methodology): “Recently, inspired by the success of generative priors in face restoration tasks (Yang et al. 2021; Wang et al. 2021a), some works exploit the priors extracted from VQGAN (Esser, Rombach, and Ommer 2021), diffusion model (Ho, Jain, and Abbeel 2020) and pre-trained T2I model (Rombach et al. 2021) to…”
Towards Robust Blind Face Restoration with Codebook Lookup Transformer (2022, neutral): “GFP-GAN [38], and GPEN [44] introduce obvious artifacts and GLEAN [3] produces over-smoothed results that lack facial details.”