Towards Robust Blind Face Restoration with Codebook Lookup Transformer

By Shangchen Zhou, Kelvin C. K. Chan, Chongyi Li, Chen Change Loy. Neural Information Processing Systems, 2022

413 citations recorded by the source corpus.

Blind face restoration is a highly ill-posed problem that often requires auxiliary guidance to 1) improve the mapping from degraded inputs to desired outputs, or 2) complement high-quality details lost in the inputs. In this paper, we demonstrate that a learned discrete codebook prior in a small proxy space largely reduces the uncertainty and ambiguity of restoration mapping by casting blind face restoration as a code prediction task, while providing rich visual atoms for generating high-quality faces. Under this paradigm, we propose a Transformer-based prediction network, named CodeFormer, to model the global composition and context of the low-quality faces for code prediction, enabling the discovery of natural faces that closely approximate the target faces even when the inputs are severely degraded. To enhance the adaptiveness for different degradation, we also propose a controllable feature transformation module that allows a flexible trade-off between fidelity and quality. Thanks to the expressive codebook prior and global modeling, CodeFormer outperforms the state of the arts in both quality and fidelity, showing superior robustness to degradation. Extensive experimental results on synthetic and real-world datasets verify the effectiveness of our method.

Subfields: Computer Vision and Pattern Recognition, Dermatology, Neurology

View Paper: https://www.semanticscholar.org/paper/5858e4c673ce8dc9d613e89eb5929fa8dc694189

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Chen Change Loy
2. Luc Van Gool
3. Yu Qiao
4. Lei Zhang
5. Chao Dong
6. Xiaogang Wang
7. Ming-Hsuan Yang
8. Wanli Ouyang

In Dermatology

1. Wanli Ouyang
2. Chen Change Loy

In Neurology

1. Yu Qiao
2. Chen Change Loy
3. Wanli Ouyang

Citations in the corpus

• DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior (2023, methodology): “However, the problem setting of zero-shot image restoration (ZIR) is not in accordance with BIR.”
• Exploiting Diffusion Prior for Real-World Image Super-Resolution (2023, positive): “To further facilitate a trade-off between re-alism and fidelity, depending on user preference, we follow CodeFormer [84] to introduce an optional controllable feature wrapping module.”
• Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild (2024, background): “Beyond GANs, other generative models can also serve as priors [10, 36, 54, 73, 97–99].”