Frequency Separation for Real-World Super-Resolution

By Manuel Fritsche, Shuhang Gu, Radu Timofte. 2019

275 citations recorded by the source corpus.

Most of the recent literature on image super-resolution (SR) assumes the availability of training data in the form of paired low resolution (LR) and high resolution (HR) images or the knowledge of the downgrading operator (usually bicubic downscaling). While the proposed methods perform well on standard benchmarks, they often fail to produce convincing results in real-world settings. This is because real-world images can be subject to corruptions such as sensor noise, which are severely altered by bicubic downscaling. Therefore, the models never see a real-world image during training, which limits their generalization capabilities. Moreover, it is cumbersome to collect paired LR and HR images in the same source domain. To address this problem, we propose DSGAN to introduce natural image characteristics in bicubically downscaled images. It can be trained in an unsupervised fashion on HR images, thereby generating LR images with the same characteristics as the original images. We then use the generated data to train a SR model, which greatly improves its performance on real-world images. Furthermore, we propose to separate the low and high image frequencies and treat them differently during training. Since the low frequencies are preserved by downsampling operations, we only require adversarial training to modify the high frequencies. This idea is applied to our DSGAN model as well as the SR model. We demonstrate the effectiveness of our method in several experiments through quantitative and qualitative analysis. Our solution is the winner of the AIM Challenge on Real World SR at ICCV 2019.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Radu Timofte
3. Chen Change Loy
4. Ming-Hsuan Yang
5. Lei Zhang
6. Dacheng Tao
7. Chao Dong
8. Yu Qiao

In Media Technology

1. Radu Timofte
2. Chao Dong
3. Luc Van Gool
4. Chen Change Loy
5. Kai Zhang
6. Jingyun Liang
7. Lei Zhang
8. Wangmeng Zuo

Citations in the corpus

• Designing a Practical Degradation Model for Deep Blind Image Super-Resolution (2021, methodology): “We compare the proposed BSRNet and BSRGAN with RRDB [49], IKC [16], ESRGAN [49], FSSR-DPED [13], FSSR-JPEG [13], RealSR-DPED [20] and RealSR-JPEG [20].”
• NTIRE 2020 Challenge on Real-World Image Super-Resolution: Methods and Results (2020, methodology): “This method is based on the DSGAN [18] approach.”
• Focal Frequency Loss for Image Reconstruction and Synthesis (2020, background): “The application areas of the frequency domain analysis have been further extended, including media forensics [59, 72, 22, 12], super-resolution [13, 63], generalization analysis [58, 20], magnetic resonance imaging [53], image rescaling [64], etc.”
• Real-World Single Image Super-Resolution: A Brief Review (2021, methodology): “[106] first bicubically downsample HR images.”
• Unsupervised Image Super-Resolution with an Indirect Supervised Path (2019, methodology): “[5] further improved this method by separating the low and high frequency components and treat them differently.”
• Real-World Super-Resolution via Kernel Estimation and Noise Injection (2020, methodology): “[10] propose the DSGAN model to generate degraded images.”
• Fast Vision Transformers with HiLo Attention (2022, background): “Based on this insight, a plethora of solutions have been proposed for image superresolution [53, 14], generalization [19], image re-scaling [44] and neural network compression [47, 5].”
• Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data (2021, methodology): “Thus, implicit modeling attempts to learn a degradation generation process within networks [52, 11, 44].”
• Exploiting Diffusion Prior for Real-World Image Super-Resolution (2023, unknown): “Building upon such degradation pipelines, recent works based on diffusion models [52,54] further show competitive performance on real-world image SR.”