Blind Super-Resolution Kernel Estimation using an Internal-GAN

By Sefi Bell-Kligler, Assaf Shocher, Michal Irani. 2019

533 citations recorded by the source corpus.

Super resolution (SR) methods typically assume that the low-resolution (LR) image was downscaled from the unknown high-resolution (HR) image by a fixed `ideal’ downscaling kernel (e.g. Bicubic downscaling). However, this is rarely the case in real LR images, in contrast to synthetically generated SR datasets. When the assumed downscaling kernel deviates from the true one, the performance of SR methods significantly deteriorates. This gave rise to Blind-SR - namely, SR when the downscaling kernel (``SR-kernel’’) is unknown. It was further shown that the true SR-kernel is the one that maximizes the recurrence of patches across scales of the LR image. In this paper we show how this powerful cross-scale recurrence property can be realized using Deep Internal Learning. We introduce ``KernelGAN’’, an image-specific Internal-GAN, which trains solely on the LR test image at test time, and learns its internal distribution of patches. Its Generator is trained to produce a downscaled version of the LR test image, such that its Discriminator cannot distinguish between the patch distribution of the downscaled image, and the patch distribution of the original LR image. The Generator, once trained, constitutes the downscaling operation with the correct image-specific SR-kernel. KernelGAN is fully unsupervised, requires no training data other than the input image itself, and leads to state-of-the-art results in Blind-SR when plugged into existing SR algorithms.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Media Technology, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Liang Wang
2. Tien-Ping Tan
3. Luc Van Gool
4. Yan Huang
5. Yu Qiao
6. Kai Zhang
7. Hanseok Ko
8. Jingyun Liang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Chen Change Loy
3. Radu Timofte
4. Ming-Hsuan Yang
5. Dacheng Tao
6. Lei Zhang
7. Yu Qiao
8. Yi Yang

In Media Technology

1. Radu Timofte
2. Luc Van Gool
3. Chao Dong
4. Kai Zhang
5. Jingyun Liang
6. Jian Chen
7. Jinjin Gu
8. Chen Change Loy

In Generative Models

1. Daniel Cohen-Or
2. Eli Shechtman
3. Radu Timofte
4. Jun-Yan Zhu
5. Luc Van Gool
6. Richard Zhang
7. Tero Karras
8. Xintao Wang

Citations in the corpus

• Real-World Single Image Super-Resolution: A Brief Review (2021, methodology): “1DIV2KRK [94] is available at http://www.”
• SRFlow: Learning the Super-Resolution Space with Normalizing Flow (2020, unknown): “A different line of research [6,39,40] exploit the internal patch recurrence by only training the network on the input image itself.”
• Unsupervised Degradation Representation Learning for Blind Super-Resolution (2021, methodology): “For example, KernelGAN conducts network training during test and takes over 60 seconds for a single image [3].”
• Real-World Super-Resolution via Kernel Estimation and Noise Injection (2020, methodology): “On one hand, we first use the existing kernel estimation method [3] to generate more realistic LR images.”
• Closed-Loop Matters: Dual Regression Networks for Single Image Super-Resolution (2020, methodology): “Recently, some blind SR methods [2, 55] were proposed to learn the unknown degradation methods.”
• NTIRE 2020 Challenge on Real-World Image Super-Resolution: Methods and Results (2020, methodology): “Downsampling The team performs downsampling on the clean HR images using the estimated kernels by KernelGAN [7].”
• Flow-based Kernel Prior with Application to Blind Super-Resolution (2021, methodology): “Since the blur kernels of real LR images are usually unimodal [35, 41] and can typically be modeled by a Gaussian [40], most existing blind SR works [3, 17, 21, 40, 42, 43, 49, 54, 55] assume the SR kernel is isotropic or anisotropic Gaussian kernel.”
• Unfolding the Alternating Optimization for Blind Super Resolution (2020, methodology): “We use KernelGAN [3] + ZSSR [29] and IKC [14] as the representative methods for blind SR, and CARN [2] as the representative method for non-blind SR method.”
• The Role of ImageNet Classes in Fréchet Inception Distance (2022, methodology): “…2020b;a; 2021), unsupervised domain translation (Zhu et al., 2017; Choi et al., 2020; Kim et al., 2020), image super resolution (Ledig et al., 2017; Bell-Kligler et al., 2019; Saharia et al., 2021), image editing (Park et al., 2019; 2020; Huang et al., 2022) and generating images based on a text…”
• Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild (2024, background): “Over time, the need for blind IR methods that are not based on specific degradation assumptions has grown [5, 10, 25, 34, 35, 46–48 , 91].”