Image Super-Resolution by Neural Texture Transfer

By Zhifei Zhang, Zhaowen Wang, Zhe L. Lin, Hairong Qi. Computer Vision and Pattern Recognition, 2019

342 citations recorded by the source corpus.

Due to the significant information loss in low-resolution (LR) images, it has become extremely challenging to further advance the state-of-the-art of single image super-resolution (SISR). Reference-based super-resolution (RefSR), on the other hand, has proven to be promising in recovering high-resolution (HR) details when a reference (Ref) image with similar content as that of the LR input is given. However, the quality of RefSR can degrade severely when Ref is less similar. This paper aims to unleash the potential of RefSR by leveraging more texture details from Ref images with stronger robustness even when irrelevant Ref images are provided. Inspired by the recent work on image stylization, we formulate the RefSR problem as neural texture transfer. We design an end-to-end deep model which enriches HR details by adaptively transferring the texture from Ref images according to their textural similarity. Instead of matching content in the raw pixel space as done by previous methods, our key contribution is a multi-level matching conducted in the neural space. This matching scheme facilitates multi-scale neural transfer that allows the model to benefit more from those semantically related Ref patches, and gracefully degrade to SISR performance on the least relevant Ref inputs. We build a benchmark dataset for the general research of RefSR, which contains Ref images paired with LR inputs with varying levels of similarity. Both quantitative and qualitative evaluations demonstrate the superiority of our method over state-of-the-art.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0268917fdb8422b8c8b3f579ef5b1f53a3a054e1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Texture Transformer Network for Image Super-Resolution (2020, methodology): “As for RefSR methods, CrossNet [43] and SRNTT [41] are two state-of-the-art methods recently, which significantly outperform previous RefSR methods.”
GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution (2020, methodology): “We assess the efficacy of the new notion of GAN-based dictionary by comparing GLEAN with two representative methods adopting an imagery dictionary for SR – SRNTT [47] and DFDNet [23].”
Language-Driven Image Style Transfer (2021, background): “Artistic style transfer has a considerable application value for creative visual design, such as image stylization and video effect [112, 113, 114, 115, 116].”
Blind Face Restoration via Deep Multi-scale Component Dictionaries (2020, methodology): “To solve this issue, reference-based methods [7, 26, 35, 47] have been suggested by using reference prior in image restoration task to improve the process of network learning and alleviate the dependency of network on degraded input.”
Correction to: Single Image Super-Resolution via a Holistic Attention Network (2020, background): “[42] further transferred textures from reference images according to the textural similarity to enhance textures.”
Exploiting Diffusion Prior for Real-World Image Super-Resolution (2023, neutral): “This involves us-ing one or several reference high-resolution (HR) images, which share similar textures with the input low-resolution (LR) image, as an explicit prior to aid in generating the corresponding HR output.”
Extended Feature Pyramid Network for Small Object Detection (2020, methodology): “Recent reference-based SR methods [35], [36] have capacity of enhancing SR images with textures or contents from reference images.”
Real-World Super-Resolution via Kernel Estimation and Noise Injection (2020, background): “To address this issue, some researchers enhanced realistic texture via spatial feature transform [47, 48, 38].”