FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors

By Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, Jian Yang. 2017

570 citations recorded by the source corpus.

Face Super-Resolution (SR) is a domain-specific superresolution problem. The facial prior knowledge can be leveraged to better super-resolve face images. We present a novel deep end-to-end trainable Face Super-Resolution Network (FSRNet), which makes use of the geometry prior, i.e., facial landmark heatmaps and parsing maps, to super-resolve very low-resolution (LR) face images without well-aligned requirement. Specifically, we first construct a coarse SR network to recover a coarse high-resolution (HR) image. Then, the coarse HR image is sent to two branches: a fine SR encoder and a prior information estimation network, which extracts the image features, and estimates landmark heatmaps/parsing maps respectively. Both image features and prior information are sent to a fine SR decoder to recover the HR image. To generate realistic faces, we also propose the Face Super-Resolution Generative Adversarial Network (FSRGAN) to incorporate the adversarial loss into FSRNet. Further, we introduce two related tasks, face alignment and parsing, as the new evaluation metrics for face SR, which address the inconsistency of classic metrics w.r.t. visual perception. Extensive experiments show that FSRNet and FSRGAN significantly outperforms state of the arts for very LR face SR, both quantitatively and qualitatively.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Image Super-Resolution via Iterative Refinement (2021, methodology): “Metric PULSE [17] FSRGAN [14] Regression SR3”
PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models (2020, background): “Some researchers have attempted to extend these MSEbased methods to additionally optimize on metrics intended to encourage realism, serving as a force opposing the smoothing pull of the MSE term [15, 7].”
Deep Unfolding Network for Image Super-Resolution (2020, background): “While learning-based blind image restoration has recently received considerable attention [12, 39, 43, 50, 62], we note that this work focuses on non-blind SISR which assumes the LR image, blur kernel and noise level are known beforehand.”
Portrait shadow manipulation (2020, background): “2017], super-resolution [Chen et al. 2018] and synthesis [Denton et al.”
Face De-Spoofing: Anti-Spoofing via Noise Modeling (2018, background): “De-X problems De-X problems, such as de-noising, de-blurring, de-mosaicing, superresolution and inpainting [33,13,14,15,34,35,16,17,36,37,38], are classic low-level vision problems that remove the degradation effect or artifacts from the image.”
Cross-Resolution Face Recognition via Prior-Aided Face Hallucination and Residual Knowledge Distillation (2019, methodology): “Chen et al [4] recover LR images with the help of geometry prior estimation, i.”
DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior (2023, background): “facial parsing maps [5], facial landmarks[9; 27], and facial component heatmaps [62]) or reference priors[34; 33; 32; 13] as auxiliary information to guide the face restoration process.”
HiFaceGAN: Face Renovation via Collaborative Suppression and Replenishment (2020, background): “For category-specific [2] (face) restoration, it is commonly believed that incorporating external guidance on facial prior would boost the restoration performance, such as semantic prior [38], identity prior [12], facial landmarks [4][5] or component heatmaps [59].”
Efficient Low-Resolution Face Recognition via Bridge Distillation (2020, methodology): “In our experiment, we check two hallucination meth-ods, BI (bilinear interpolation) and SR (a recent FSRNet approach [59]), as well as five high-resolution face recognition models (CenterLoss [58], SphereFace [3], CosFace [5], VGGFace2 [6] and ArcFace [31]).”
Towards Real-World Blind Face Restoration with Generative Facial Prior (2021, background): “Previous works [9, 70, 6] typically exploit face-specific priors in face restoration, such as facial landmarks [9], parsing maps [6, 9], facial component heatmaps [70], and show that those geometry facial priors are pivotal to recover accurate face shape and details.”
Real-World Super-Resolution via Kernel Estimation and Noise Injection (2020, background): “In recent years, deep learning-based methods [9, 35, 7, 34, 20, 19, 23] have achieved remarkable results with respect to ﬁdelity performance, which mainly focuses on designing network structures to further improve the performance of speciﬁc datasets.”
Blind Face Restoration via Deep Multi-scale Component Dictionaries (2020, background): “to measure the similarity between 8 Xiaoming Li, et al. the Fd s;cand all the clusters in RDic . For k-thcluster in component dictionary, the similarity is dened as: Sk s;c = D Fd s;c;RDic k s;c E ; (4) The input component feature Fd s;cmatches across all the clusters in the re-normed component dict”
Towards Robust Blind Face Restoration with Codebook Lookup Transformer (2022, unknown): “Various priors have been used to mitigate the ill-posedness of this problem, including geometric priors [4, 5, 29, 43], reference priors [22–24], and generative priors [2, 36, 42].”
Generative Diffusion Prior for Unified Image Restoration and Enhancement (2023, background): “1 [11] Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian Yang.”
PSCC-Net: Progressive Spatio-Channel Correlation Network for Image Manipulation Detection and Localization (2021, background): “It has been widely adopted in many low-level and high-level vision tasks, such as denoising [5], [49], inpainting [50], super-resolution [51], [52], and object detection [53]–[56].”
DSFD: Dual Shot Face Detector (2018, background): “Face detection is a fundamental step for various facial applications, like face alignment [26], parsing [3], recognition [34], and veriﬁcation [6].”
To learn image super-resolution, use a GAN to learn how to do image degradation first (2018, methodology): “FSRNet provides only testing code (trained on their dataset using pairs of bilinearly downsampled - HR images).”
Deep Learning for Image Super-Resolution: A Survey (2019, methodology): “And the FSRNet [73] uses not only facial landmark heatmaps but also face parsing maps as the facial prior constraints.”